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Section 1: Overview 


One of the broad intents of the Elementary and Secondary Education Act (ESEA) as 
amended by the Every Student Succeeds Act (ESSA) is to encourage evidence-based 
decision-making as a way of doing business. Nonregulatory guidance issued in September 
2016 by the U.S. Department of Education (ED) clarifies and expands on both the nature of 
evidence-based improvement and the levels of evidence that are specified in the law. This 
guide builds on that ED guidance and provides an initial set of tools to help states and 
districts understand and plan for implementing evidence-based improvement strategies. 

This guide recognizes school and district improvement as a continuous, systemic, and 
cyclical process, and emphasizes the use of evidence in decision-making throughout 
continuous improvement. In other words, the guide is not aimed at isolated decisions; rather, 
it is meant to support evidence-based decision-making (especially selection of interventions) 
that is nested within a larger improvement process. See section 2 for more on this important 
point. 


Audience 

The primary audience for this guide is state education agency (SEA) staff who are 
responsible for understanding and implementing the evidence-based provisions of ESSA. 
Because SEAs vary widely in their organizational structures and in the titles of different 
positions, this guide refers generally to SEAs or SEA staff, leaving to each state and its 
technical assistance providers, such as Regional Comprehensive Centers (RCCs) and 
Regional Educational Laboratories (RELs), the determination of who should be engaged. 

While the guide may be used within a single SEA department or program, it can also 
contribute to alignment of cross-agency interventions, consistent messaging, and a unified 
approach to supporting school improvement efforts across the SEA. The ideal audience is a 
team of SEA staff representing multiple programs or departments. SEAs may also want to 
engage their intermediary networks, and technical assistance providers charged with 
directing, guiding, supporting, and monitoring districts, to select and implement evidence- 
based practices as defined in ESSA. 

Additionally, some of the tools in this guide have been designed to be used by local 
education agencies (LEAs), either directly or with facilitation by SEAs or technical assistance 
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providers. Therefore, a secondary audience consists of the LEAs within a given state, 
including district leadership teams and/or district department heads. 


Purpose 

The purpose of the guide is to build capacity of SEAs and their intermediaries to support 
LEAs in understanding the evidence-related requirements of ESSA and, consequently, 
selecting and implementing interventions that are evidence-based and that have strong 
potential to improve student outcomes. Specifically, the guide is intended to: 

1) increase readers’ understanding of the expectations and opportunities for evidence- 
based school and district improvement in the context of ESSA; 

2) encourage a broad understanding of the elements of evidence-based decision- 
making, including how needs, context, implementation strategies, desired outcomes, 
and sustainability considerations inform choices of evidence-based interventions, and 
how formative and summative evaluation are integral to an evidence-based 
improvement cycle; and 

3) offer guiding information and a starter set of six tools to support this work, with an 
emphasis on the process of selecting evidence-based interventions. 

The materials presented in the guide offer SEAs and their LEAs opportunities to conduct a 
review of their approach to school and district improvement, including selection of evidence- 
based interventions, and to develop action steps for strengthening the guidance and supports 
that SEAs offer to their LEAs and that LEAs offer to their schools. 


Contents 

In addition to this section, the following four sections provide further background, tools, and 
additional resources. 

• Section 2 includes further discussion of the context and requirements of ESSA in 
relation to evidence-based decision-making, and describes a framework tor a 
continuous improvement process grounded in evidence-based decision-making. 

• Section 3 gives suggestions on how to use the tools in the guide, including 
information about facilitation strategies and options for modifying the tools to fit state 
and local contexts. This section also emphasizes the importance of preparing for 
using the tools. 

• Section 4 provides six tools, each designed to encourage focused conversations and 
support cross-agency collaboration. The first two tools guide examination of state and 
district improvement and decision-making frameworks. The second two tools help 
SEAs and LEAs explore strategies for providing guidance on selecting evidence- 
based interventions. The last two tools support selection of evidence-based 
interventions. 
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• Section 5 offers a list of additional resources to further the conversation, and 

enhance the work, initiated by this guide. This section includes examples of publicly 
available tools for evidence-based improvement, and sources for research and 
information on evidence-based interventions. 

This guide was specifically designed to be a starting point for making evidence-based 
decisions, and is not intended to be comprehensive. It contains initial information and tools to 
guide conversations and foster deeper thinking around evidence-based decision-making, 
especially within an improvement process. Therefore, we encourage the use of this guide as 
an organizer for collecting or developing other tools and resources that more deeply explore 
particular steps that are not explicitly covered in this guide. For example, tools 3-6 in this 
guide focus on a portion of the second step of the continuous improvement process 
(examining evidence to select an intervention). This is largely due to the fact that this step is 
more closely informed than the other steps by the ESSA evidence-based provisions and ED 
guidance; thus, it is the main focus of this guide. However, additional tools that focus on other 
steps in the process exist and could be added to complement this guide. One way this might 
be accomplished is through tools 1 and 2. SEAs can use tool 1 to reflect on a state’s 
frameworks and tools related to the entire continuous improvement cycle and on what might 
need to be refined or added to best reflect that cycle. Where gaps exist, RCCs, RELs, or 
others can develop new tools, in the future, to round out the existing set of tools. See table 1 
for a list of tools included in the guide, as well as their respective audiences and linkages to 
stages in the improvement cycle. 

This guide will be revised based on feedback from initial users. Not only do we espouse 
continuous improvement, but we intend to practice it. Examples of how SEAs and LEAs have 
used, adapted, and refined the tools can enrich subsequent versions of the guide and can 
support an exchange of strategies across states. We invite your participation. 


Table 1 . Initial Set of Tools Provided in This Guide 


# 

Tool Title 

Primary 

Audience 

Improvement 
Cycle Step 1 

1 

SEA Inventory of Current Practice 

SEA 

All 

2 

LEA Inventory of Current Practice 

LEA 

All 

3 

SEA Guidance for Evidence-Based Interventions 

SEA 

Select 

4 

LEA Guidance for Evidence-Based Interventions 

LEA 

Select 

5 

Intervention Evidence Review 

SEA or LEA 

Select 

6 

Comparing Evidence-Based Interventions 

SEA or LEA 

Select 


1 See figure 1 on page 8 for improvement cycle steps. 


Section 1: Overview | 3 








Section 2: Context and Framework 


Context for This Guide 

Evidence-based decision-making has a long history in fields outside of education, such as 
medicine. More recently, recipients of federal funding in other fields have been required to 
adopt interventions that are evidence-based. Federal education funding is no exception. 1 The 
term “evidence-based” is used repeatedly in the Every Student Succeeds Act (ESSA), the 
latest reauthorization of the Elementary and Secondary Education Act (ESEA). 11 ESEA 
previously referred to “scientifically based research”; the term “evidence-based” is more 
strictly defined. ESSA calls on states, districts, and schools to select evidence-based 
activities, strategies, or interventions (collectively referred to in this guide as “interventions”). 
ESSA defines four levels of evidence, which are further described in the nonregulatory 
guidance issued by the U.S. Department of Education (ED) in September 2016. 

• Strong evidence — demonstrates a statistically significant effect on improving student 
outcomes or other relevant outcomes, based on at least one well-designed and well- 
implemented experimental study. 

• Moderate evidence — demonstrates a statistically significant effect on improving 
student outcomes or other relevant outcomes, based on at least one well-designed 
and well-implemented quasi-experimental study. 

• Promising evidence — demonstrates a statistically significant effect on improving 
student outcomes or other relevant outcomes, based on at least one well-designed 
and well-implemented correlational study with statistical controls for selection bias. 

• Demonstrates a rationale — demonstrates a rationale based on high-quality research 
findings or positive evaluation that such intervention is likely to improve student 
outcomes or other relevant outcomes; and includes ongoing efforts to examine the 
effects of the intervention. 

Some programs and funding streams allow the use of all four levels; others, notably the set- 
aside for low-performing schools in Title I, do not allow the use of the fourth level of evidence. 
See ESSA and other resource documents listed in section 5 for additional information on 
these levels of evidence. 
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Beyond defining four levels of acceptable evidence, the law provides states with more 
flexibility and authority, compared to what was allowed under No Child Left Behind."' 
Regarding how states and districts handle selecting and implementing interventions, §200.21 
of ESSA requires a state to review and approve each comprehensive support and 
improvement plan in a timely manner. Further, the regulations require the state education 
agency (SEA) to monitor and periodically review each local education agency (LEA)’s 
implementation of its plan. §200. 23(c)(2) allows a state to establish an exhaustive or non- 
exhaustive list of state-approved, evidence-based interventions for use in schools 
implementing comprehensive or targeted support and improvement plans. 

The provisions in ESSA also lend themselves to the use of an iterative, continuous 
improvement process. The law specifies that states are to continuously evaluate the 
effectiveness of interventions carried out under several federal grant programs (e.g., ESSA, 
2015, Section 4624[10]). Finally, regulations of ESSA (24 C.F.R. § 200.23, 2016) require 
states to evaluate the effects of evidence-based interventions on student achievement and 
other outcomes, and to disseminate the results of those evaluations to LEAs. The intention of 
these requirements is to expand the evidence base for interventions, including studies in a 
variety of contexts. 

Section 5 of this guide includes sources of further information about ESSA and its evidence 
provisions. 


Framework for Using Evidence in a Continuous Improvement Process 

The five-part continuous improvement framework described in this guide (see figure 1 on 
page 8) brings together two important ideas: (1) decisions should be based on data and 
evidence, and (2) a continuous improvement process is fundamental to engaging in and 
sustaining improvements in school and district practice. ESSA provides SEAs with 
opportunities to establish and support authentic continuous improvement processes. iv 
Accordingly, the framework deliberately outlines school and district improvement as a 
continuous, systemic, and cyclical process. This guide is meant to support the use of 
evidence in making decisions that are nested within this framework of a continuous 
improvement process, an ongoing process that is larger than any isolated decisions that 
schools or districts make to improve their practice. 


“A continuous improvement process starts with the 
problem, rather than the solution.” 


The literature on decision-making in education reveals an array of factors that often influence 
decisions, including popular trends, political considerations, and the networks and information 
sources with which decisionmakers are connected. v ESSA and, more generally, the 
evidence-based decision-making movement emphasize the importance of evidence in 
informing decisions. Knowing and building on what has worked in the past, and specifically 
for whom and in what circumstances, offers a better chance of success in the future. 
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However, overfocusing on the decision itself can perpetuate a “magic bullet” concept of 
improvement: the fact that a program produces positive outcomes on average does not mean 
that it will do so in every case. Deciding to implement a particular approach must be 
preceded by a thorough assessment of needs and hypotheses about the causes of issues 
and problems, to determine if a proposed program or practice is really appropriate and what 
adaptations may be necessary, and it must be followed by careful implementation and 
analysis of local outcomes. 


“Using data and evidence keeps the improvement process 
guided toward the desired outcomes.” 


A continuous improvement process starts with the problem, rather than the solution. vi It 
includes addressing a discrete issue or problem by systematically testing potential solutions 
while tracking well-defined and measurable goals. The process is meant to be iterative — data 
are collected, analyzed, and discussed frequently so that adjustments can be made to the 
intervention or program, and then data are collected and analyzed once again. In addition, 
the scale of the initial effort often begins small and expands overtime as the intervention is 
refined. Using this process, schools and districts often start with a pilot intervention or activity 
and expand it as the fit to local conditions is better understood. 

Continuous improvement cultivates a problem-solving orientation and close observation of 
the system that is producing the outcomes. vii This orientation is important to sustained 
improvement, especially when more than one change may be needed. Using data and 
evidence keeps the improvement process guided toward the desired outcomes. 


“Evidence-based decision-making and reflection are the 
core of the entire continuous improvement process and 
are used in each step.” 


The framework of five fundamental steps shown in figure 1 conveys the elements that are 
present in every strong continuous improvement process. These steps mirror those in the ED 
guidance of September 2016, with some differences in terminology. It is not our intent to 
suggest that all models must use these same five steps — some versions of continuous 
improvement processes include more or fewer steps — but the five steps shown in figure 1 
convey the intent of a full cycle. Resources for more information about continuous 
improvement processes can be found in section 5 of this guide. 
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Figure 1. Evidence-Based Improvement 
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Evidence-based decision-making and reflection are the core of the entire continuous 
improvement process and are used in each step. The steps overlap, with each leading into 
the next, so that, for example, the Analyze step begins before the Implement step is 
completed; the color shading in figure 1 is intended to communicate this point. 

The remainder of this section briefly summarizes each step. The descriptions and guiding 
questions in the ED guidance of September 2016 are another source of input. Tools 1 and 2 
(in section 4 of this guide) are designed for SEAs and LEAS to reflect on how these steps 
may occur in their respective models. 

Step 1 : Inform — The first step is to analyze the needs of the education setting, in order to 
inform subsequent steps, particularly decisions that are made in step 2. Needs are analyzed 
by using input from as many stakeholders as possible: leadership, staff, parents and other 
community members, and students. The needs assessment data are used to identify and 
prioritize gaps in the educational setting, whether they are programmatic or service- or staff- 
related. Well-defined and measurable goals are developed from a careful analysis of these 
needs and gaps, and from hypotheses about which factors in the current situation might be 
causing problems and impeding attainment of desired outcomes. 
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Step 2: Select — This step involves identifying, examining, and selecting effective programs 
or practices for the intended setting and population(s). The step might start with searching 
clearinghouses of evidence-based interventions, such as the What Works Clearinghouse 
(WWC), which has reviewed the research on many interventions (see section 5 for additional 
clearinghouses to consider). States may then suggest or require specific interventions, 
depending on local policies, from lists of evidence-based interventions. In addition, states 
may want to disseminate research on local interventions that has not yet been included in 
national clearinghouses, but they should first obtain an assessment of the rigor of the 
research. RELs can help provide these assessments. See section 5 for REL contact 
information. Careful attention to the quality of both individual research studies and the body of 
evidence on an intervention is needed. 

Selection also includes taking stock of the specific context and educational environment(s) in 
which an intervention will be implemented, including the student population and the local 
capacity, resources, and strategic plans. What works in one place will not necessarily work in 
another. The results of this step provide the specifics needed to develop detailed 
implementation plans. 

Step 3: Plan — In this step, a detailed implementation plan is developed for the selected 
interventions, to specify who will implement the interventions, when, and with what support. 
Planners determine what core features are needed for implementation with fidelity, and what 
adaptations may be needed. Also, necessary materials, technical assistance, and 
professional development for the actual implementation are either developed or contracted. 
Plans for analysis and/or evaluation are drafted, and data are collected to monitor progress. 

Step 4: Implement — This step involves carrying out the intervention on a small or large 
scale, depending on the maturity of the intervention. Educators might start small (e.g., a 
single class in a grade; one grade in a school; one school in a district) and then expand later. 
It is important for this step to include the collection and examination of implementation data 
for formative feedback and improvement. Educators will need to ensure that the interventions 
are being implemented as was planned in the previous step, and will need to correct 
problems (e.g., teachers not participating in the intended level of professional development) 
and document any promising adaptations that might be informative to others. Implementation 
is continually assessed in this step, through an iterative process, until the intervention is 
being delivered in a stable way. 

Step 5: Analyze — In this step, data are collected about longer-term changes in primary 
outcomes. If there is progress toward the goals, the intervention can be continued and 
expanded when appropriate. If not, a new or additional strategy may be needed. As laid out 
in the ED guidance of September 2016, this step may involve progress monitoring — tracking 
trends in outcomes overtime. Or, if an intervention is stable enough, a rigorous evaluation of 
impact may be appropriate. Finally, the findings from this step can be communicated 
outward; therefore, the entire community can benefit, as reflected in the ESSA requirement 
that states share evaluation information. 
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Section 3: Making the Most 
of This Guide 


Overall Use 

This guide includes initial considerations and a starter set of tools to help state education 
agencies (SEAs) as they: 

1) make explicit how they currently support local education agencies (LEAs) in 
evidence-based improvement; 

2) reflect on that support to ensure that it is appropriately aligned with Every Student 
Succeeds Act (ESSA) expectations and opportunities; and 

3) consider actions to enhance their support. 

Some tools in the guide are designed for LEAs, with similar purposes of explication, 
reflection, and enhancement. 

Completion of the tools should not be a goal in and of itself; use of this guide is not a 
checklist exercise. Instead, we encourage using the tools as conversation starters. Working 
through the various prompts in the tools creates artifacts or documentation of these 
conversations, which can lead to further discussion. 


“Completion of the tools should not be a goal in and of 
itself; use of this guide is not a checklist exercise. 
Instead, we encourage using the tools as conversation 

starters.” 


The set of tools provided in this guide does not address every step of the improvement 
process outlined in section 2. The first two tools support examination of the whole evidence- 
based improvement cycle; the other four tools focus on the second step of the improvement 
process shown in figure 1 on page 8 (Select). Several ESSA-derived considerations are 
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particularly relevant to this second step. An SEA may want to start with tool 1, the inventory 
of current practice in evidence-based improvement, and then consider when and how to use 
the other tools, either as an SEA or with LEAs. The tools are designed so that each may be 
used as a stand-alone activity. However, greater benefit can likely be derived by using the 
tools in combination. 

We encourage cross-agency teams of SEA staff to work through this guide together, so that 
the state's approach includes the different perspectives, needs, and roles across all of the 
SEA programs or departments that are involved in school improvement efforts. This approach 
is an opportunity to invite and engage multiple staff who can move the work forward.™ Each 
SEA will likely have a leadership team or designated lead for ESSA planning, including 
planning around the evidence-based provisions. This leadership group may want to identify 
key stakeholders across the agency, share this guide with them, and identify a first tool for 
the group to work with together. 

Additionally, several of the tools (tools 2, 4, 5, and 6) are designed so that they may be used 
by LEAs. Therefore, similar to the SEA process, LEAs may want to identify a cross-district 
team to work with the tools and, if appropriate, engage with their SEA or intermediary service 
provider. 

The tools are available as Word documents so that they can be modified to fit local needs 
and contexts. For example, an SEA may want to engage LEAs in reviewing the evidence- 
based improvement process at the local level (tool 2), but may want to first make changes in 
the tool 2 form to reflect the state’s own framework and terminology. The tools are intended 
to encourage evidence-based improvement, not to require the particular language or 
structure used in the framework in this guide. To access the Word versions of the tools go to: 
http://www.wested.orq/evidence-based-improvement-essa-quide-tools-for-states. 


Overview of Tools 

This section provides a brief description of each tool and expected outcomes of using each 
tool. Each tool provided in section 4 includes a description of purpose, intended outcome 
(also listed in table 2 below), materials needed, recommended participants, time commitment, 
detailed instructions, guidelines for leading conversations, and possible modifications or 
variations. 

Tool 1 : SEA Inventory of Current Practice — This tool is designed to guide an SEA team 
to make explicit the state’s framework for improvement planning and how the SEA currently 
supports LEAs in making evidence-based decisions within that framework, and to reflect on 
whether this support meets the ESSA requirements and expectations. This tool can help 
SEAs to identify needed changes or adjustments to the structure of an SEA's evidence-based 
continuous improvement process, in order to ensure that the process is comprehensive and 
incorporates research or data throughout. The tool can aid in identifying priorities and 
necessary resources (time, funding, personnel), as well as methods of communicating with 
LEAs and stakeholders about needed changes. 

Tool 2: LEA Inventory of Current Practice — Similar to tool 1 , this tool is designed to help 
SEAs and their intermediary technical assistance networks to guide LEA teams to make 
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explicit the LEA's framework for improvement planning and how the LEA currently supports 
its schools in making evidence-based decisions within that framework, and to reflect on 
whether this support meets the ESSA requirements and expectations. An LEA may decide to 
work independently on this tool after the SEA has provided initial direction. This tool can help 
identify needed changes or adjustments to the structure of an LEA's evidence-based 
continuous improvement process, in order to ensure that the process is comprehensive and 
incorporates research or data throughout. The tool can aid in identifying priorities and 
necessary resources (time, funding, personnel), as well as methods of communicating with 
schools and stakeholders about needed changes. 

Tool 3: SEA Guidance for Evidence-Based Interventions — This tool will assist SEAs in 
defining or delineating their approaches to guiding LEAs in selecting evidence-based 
interventions. The SEA's approach may include selecting specific interventions (e.g., a list 
that LEAs might choose from) or compiling resources (e.g., a list of resources that LEAs may 
explore to identify interventions). 

Tool 4: LEA Guidance for Evidence-Based Interventions — Similar to tool 3, this tool 
focuses on specific guidance from the district to its schools with regard to selecting evidence- 
based interventions. 

Tool 5: Intervention Evidence Review — This tool is designed to assist SEAs, LEAs, or 
schools as they review research on interventions that target a specific issue, problem, or 
outcome. By completing this tool, the user can assess the levels of evidence for a specific 
intervention under consideration for selection (based on the evidence definitions in ESSA and 
nonregulatory guidance). 

Tool 6: Comparing Evidence-Based Interventions — This tool is designed for SEAs, 

LEAs, or schools to compare how different evidence-based interventions align with the 
context of a specific district or school. This tool can be used after completing tool 5, or without 
completing tool 5 if users already have sufficient information about the levels of evidence for 
specific interventions under consideration for selection. Completing this tool provides the user 
with this information across different interventions that target the same problem, issue, or 
outcome. Reviewing the information in the inventory will help in selecting an evidence-based 
intervention appropriate for the user’s context. This tool should be thought of as a guide, and, 
as such, the importance of the questions in this tool may vary for different users. 


Table 2. Expected Outcomes of Tools 


# 

Tool Title 

Outcome 

1 

SEA Inventory of Current Practice 

Identification of gaps in the SEA continuous 
improvement process 

2 

LEA Inventory of Current Practice 

Identification of gaps in the LEA continuous 
improvement process 

3 

SEA Guidance for Evidence-Based 

Interventions 

Guidance for LEAs on selecting interventions 
(e.g., from a list or vetting process) 


Section 3: Making the Most of This Guide | 13 






# 

Tool Title 

Outcome 

4 

LEA Guidance for Evidence-Based 

Interventions 

Guidance for schools on selecting interventions 
(e.g., from a list or vetting process) 

5 

Intervention Evidence Review 

A summary of the body of evidence for an 
intervention 

6 

Comparing Evidence-Based Interventions 

A summary of the feasibility of implementing 
an intervention in the local context 


Examples of Uses 

SEAs and/or LEAs can use many potential combinations of tools, depending on the SEA's 
and/or LEA’s goals and context. This section provides a few examples of possible scenarios 
for using the tools in this guide. 

SEAs and LEAs Partner on Using Tools 1-6 to Build Capacity for Evidence-Based 
Decision-making — An SEA wants to help build the capacity of its lowest-performing LEAs in 
using evidence for decision-making. The SEA's School Improvement division works with a 
group of the 10 lowest-performing LEAs in a cohort improvement academy. After using tool 1 
to document its overall continuous improvement framework and tool 3 to draft its approach to 
selecting interventions, the School Improvement team shares the results of these tools with 
the LEAs in the academy. During an academy session, the SEA introduces tools 2 and 4. 
Before their next academy session, LEAs will work through tools 2 and 4 on their own and 
submit the results to the SEA for feedback. In subsequent academy sessions, teams of LEAs 
will work together to complete tools 5 and 6 for a set of interventions that they are interested 
in assessing. The LEAs share the results of their inquiry with schools and encourage them to 
use some combination of identified interventions that meet the school’s particular context and 
needs. 

SEA Uses Tools 3, 5, and 6 to Update Intervention List — An SEA with a previously 
devised list of interventions would like to update and revise its list in light of ESSA 
requirements. After using tool 3 and reaffirming that the SEA wants to provide a list of 
interventions to LEAs, the SEA research team uses tools 5 and 6 to assess existing and 
possible interventions and shares the updated intervention list with LEAs and schools. 

SEA Cross-Agency Collaboration on Using Tools 1 and 3 to Inform LEA Plan 
Submission — A cross-agency team of SEA leaders who work with Title I, Title II, and Title IV 
funds wants to devise an agencywide approach to evidence-based interventions, to inform 
LEA plan submission. Each department leader completes tool 1 for his or her department, 
and, with the support of the SEA’s Regional Comprehensive Center, all of the department 
leaders come together to compare their tool 1 results and collectively complete tool 3. This 
cross-agency team shares the resulting framework from tool 1, and the approach defined in 
tool 3, with LEAs in a webinar. The SEA invites LEAs to complete the remaining tools in this 
guide (tools 2, 4, 5, and 6) on their own as preparation for their LEA plan submission. 
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SEA Uses Tools 1 and 3 to Devise Vetting Process for LEA Title I Plans — An SEA is 
unsure if it wants to create a list or a vetting process for LEA plans related to the use of Title I 
funds. The SEA team completes tool 1 and tool 3, deciding that it will not provide a list of 
interventions. Instead, the SEA will share a vetting process with LEAs, outlining the criteria 
that the SEA will use to approve LEA plans. To support LEAs in its region, a county office of 
education creates a working group of LEAs who are interested in using tools 5 and 6 to 
assess the body of evidence for a set of interventions. The county office contacts its local 
REL for assistance. Some of the LEAs in this group use tool 2 and tool 4 and, as a result, 
decide to share a list of interventions with their schools. Other LEAs share resources for 
vetting possible interventions at the school level; they are particularly interested in tool 6, 
which guides thinking about implementation in the local context. 


Preparation 

For this guide to be most useful, we recommend that you begin by building deep 
understanding, not only of the new ESSA legislation but also of the implications of ESSA for 
your state agency. Specifically, consider what changes may need to be made to your state's 
theory of action around school improvement and related systems of support. Therefore, 
before delving into the tools, spend some time becoming familiar with the evidence guidance 
provided for ESSA and with the specific requirements of your state's programs and funding 
sources. Section 5 contains a number of references and links for more information on this 
guidance and requirements. 

The state context is also an important consideration. State policies may set requirements for 
evidence use, or for school improvement and accountability, that shape the state’s 
framework, or that use particular language that local educators will find familiar. Preparation 
for using the tools in this guide should include gathering materials or including SEA 
participants in the process who are well versed in the relevant state context. 


General Facilitation 

Although it is not required, a skilled facilitator should be engaged to help guide an SEA team 
through the tools and resulting next steps. A facilitator can help participants examine 
assumptions, resolve differing viewpoints, and allow all team members to participate. SEAs 
may turn to the Regional Comprehensive Centers and Regional Educational Laboratories to 
engage a facilitator. The tools are not intended to be completed in a lock-step manner. 

Rather, they are designed to prompt and guide substantive conversations. Note, however, 
that if LEAs will be using some of the tools, it may be necessary to complete the SEA process 
before LEAs can begin. For example, an SEA may want to modify tool 2 to reflect the state’s 
specific improvement framework before LEAs use it. 

Working through the prompts in the tools should result in artifacts that inform next steps in a 
continuous improvement process. While each tool includes specific details for facilitating 
conversations, following is a set of general facilitation recommendations to guide the use of 
all of the tools: 
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Engage a facilitator. 


• Gather and organize materials and other documentation to inform and support the 
process of using the tools; consider a central filing system (electronic and/or hard 
copy as appropriate). 

• Develop a reasonable timeline for working through the tools and follow-up activities 
(e.g., weekly sessions over a two-month period, or two full-day sessions over the 
course of two weeks) as well as for conducting progress reviews. 

• Set norms and expectations for how the group will interact, including in consensus 
building and decision-making. 

• Ensure common understanding of the purpose and outcome of each of the tools. 
Allow adequate time for preparation before moving forward with the tools. 

• Develop a common understanding of key words or phrases used in the tools. 

• Focus on guiding meaningful dialogue by using the questions provided in each tool. 

• Establish a comfortable space and work environment (e.g., room size, adequate 
supplies). 

• Set manageable priorities, rather than “pie in the sky” ideals. 

• Facilitate task management by identifying roles and responsibilities. 

• Specify how the outcomes or products of a tool will be captured in an artifact; 
consider an online, editable document that group members can work on together. 

• Determine what communication practices are needed to keep relevant stakeholders 
informed of progress and recommendations. 


General Modifications or Variations 

The tools can be used individually or in different combinations, as well as adapted to a state 
or local context. Examples of how states have used these tools will be collected and shared 
in the future. Following are some additional considerations for modifications or variations: 

• For tools designed for LEAs, SEAs might provide guidance or training for LEA staff to 
use the tool on their own, or might identify organizations in the state system of 
support, or intermediate agencies such as counties or regional offices, that might 
work with the SEA to facilitate local discussions. 

• Smaller LEAs with fewer department-level staff may need to think differently about 
who should participate. Because these tools are designed to be a structured 
conversation, having a team increases the diversity of ideas and, thus, the value of 
the tool. Smaller or rural LEAs could include site administrators and/or teacher 
leaders. LEAs could collaborate with other districts or county office staff to build 
teams that reflect different experiences and expertise. 
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• Consider the value of accessing and reviewing graphics/models and other 
information from multiple departments, states, or organizations to inform the work. 

• If multiple programs or departments are working together on a given tool, consider 
having each program or department complete the tool separately and then using a 
consensus-building process to share, compare, and come to agreement. 

• Questions in a given tool may be skipped or refined, or questions may be added, to fit 
a specific context. In other words, modify the tool(s) to fit the context. 
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Section 4: Tools 


This section includes six tools, as follows: 

❖ Tool 1 : SEA Inventory of Current Practice 

❖ Tool 2: LEA Inventory of Current Practice 

❖ Tool 3: SEA Guidance for Evidence-Based Interventions 

❖ Tool 4: LEA Guidance for Evidence-Based Interventions 

❖ Tool 5: Intervention Evidence Review 

❖ Tool 6: Comparing Evidence-Based Interventions 
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Tool 1: SEA Inventory of 


Current Practice 


Purpose 


To help state education agency (SEA) staff members be explicit about the SEA's process for 
engaging with local education agencies (LEAs) in evidence-based improvement planning and 
decision-making, and the support the SEA provides for effective use of the process; identify 
gaps or needed changes to strengthen the model or framework; and prioritize next steps. 


Outcome 


The discussions prompted by this tool will lead to a plan of action to address gaps and 
ensure that the state’s process for supporting LEAs is evidence-based and is aligned with 
ESSA requirements. The completed tool will also serve as an artifact of discussions and 
decisions. 


Materials 


In preparation, gather guides, graphics, and other documentation that describe your state’s 
processes for improvement planning, monitoring, decision-making, and related activities. 

Who Should Be Involved 

Staff who oversee the state’s improvement efforts are key participants in working through this 
tool. Those involved should have substantive knowledge of the relevant federal programs and 
requirements. Participants may include multiple staff from one program or department, or, 
ideally, staff from multiple programs and offices, in order to support alignment across the 
agency. 


Time 


Set aside 1-2 hours for each step, or longer (3 or more hours) if multiple programs or 
departments are working together. This time can be spaced out over several weeks, with 
breaks between steps to reflect or to gather additional information to inform the subsequent 
step(s). 
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Instructions Overview (detailed instructions are provided with each step) 


❖ Step 1: Get organized, including identifying roles and responsibilities, gathering 
materials, establishing a schedule, and reviewing documentation. 

❖ Step 2: Describe the context of your SEA’s process for engaging with LEAs in 
evidence-based improvement planning and decision-making, and for providing 
support to the LEAs in this process. 

❖ Step 3: Map the steps of your improvement process to the steps of the recommended 
process (described in section 2 of this guide) and identify any gaps or needed 
changes in order for your state’s process to incorporate the elements of the 
recommended process. 

❖ Step 4: Identify the supports offered to LEAs (e.g., webinar, guidebook, training, side- 
by-side coaching, regional forums); the timing or sequence (e.g., specific dates, or 
times of the year, such as fall or spring); and any gaps or needed changes (e.g., 
redundancies, timing issues, lack of resources) in order for the SEA's process to 
incorporate the elements of the recommended process. 

❖ Step 5: Review your inventory of current practice (steps 2-4), select priority areas to 
address, and describe next steps. Develop an action plan. 

Leading the Conversation 

❖ One outcome of this conversation is to build a common understanding of an 
evidence-based framework that will drive improvement work. Especially in steps 2, 3, 
and 4, be aware that participants will likely have different backgrounds and 
experiences through which they view the framework and how it is implemented. 

While participants may seem to agree, it will be important to regularly check for 
understanding. Ask participants to repeat what they heard. Paraphrase contributions 
to ensure that your understanding is accurate. Regularly ask participants if they have 
questions. 

❖ For step 3, consider having copies of the continuous improvement framework from 
section 2 of this guide, including the descriptions of each element of the process, 
available. Being able to compare the state’s recommended framework with this 
guide’s framework will inform the conversation about alignment and possible gaps. 

Modifications or Variations 

❖ For step 3, consider using this tool in two phases. In the first phase, focus exclusively 
on the model or framework for improvement planning for your state and on identifying 
its strengths and areas for needed adjustments. In the second phase, revisit each of 
your steps and delve deeper into how the use of evidence and data is infused in your 
planning framework. Is this use of evidence an explicit activity, such as requiring 
certain data in a needs assessment or requiring an evidence base for interventions? 
What decisions does it inform? Are relevant data readily available? Are there 
adequate supports built in? You might consider modifying the provided table by 
adding a column to capture details about the use of evidence or data. 
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STEP 1 : Get Organized 


Take time to get organized and build your knowledge base by gathering necessary materials 
and building a well-informed team to complete subsequent steps. Things to consider: 

Who is involved? Identify which departments or programs should be represented; then 
identify the individuals who will participate in this work. What are their respective roles and 
responsibilities in the group (e.g., note taker, facilitator)? 



What do we need to know? Gather and organize documents and other materials that 
describe the planning and related support processes to LEAs. Consider the value of having 
graphics or models from sources other than your own department or program. Review 
documentation and ensure that all participants have strong foundational knowledge of the 
information. Does everyone have a common understanding of the subject matter (e.g., state 
improvement framework, ESSA requirements, evidence-based interventions)? 
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Attend to logistics. What will the meeting schedule be? Where will you meet? What 
materials do you need (e.g., chart paper, sticky notes)? 


Notes 
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STEP 2: Describe Context 


Briefly describe the context of your SEA's process for engaging with LEAs in evidence-based 
improvement planning and decision-making, and for providing support to the LEAs in this 
process. Questions to consider: 

❖ What model or framework have you defined that guides this work with LEAs? 

❖ What services, resources, or other supports are you required to provide to LEAs 
(e.g., based on funding or other programmatic requirements)? 

❖ How is evidence-based decision-making part of this process? Where does it fit in? 

❖ What is produced as a result of going through the improvement planning process? 


Briefly Describe Your Improvement Planning and Decision-making Processes 


Section 4: Tools | 25 


STEP 3: Map Current Process 


The first column of the table below outlines the five steps of a recommended evidence-based 
improvement process, including key decision-making elements (full descriptions of each step 
are provided in section 2 of this guide). In the second column, map the steps in your state’s 
improvement process for LEAs to the steps in the recommended process. Be sure to include 
how each step uses evidence. Note that there may be more than one step in your process for 
each step in the recommended process (e.g., the Inform step of the recommended process 
might include both “Needs Assessment” and “Goal Development” from your state’s process). 
In the third column, identify any gaps or needed changes in order for your state’s process to 
incorporate the elements of the recommended process. Questions to consider: 

❖ Does your state’s process address all of the steps in the recommended process? If 
not, what are the gaps? 

❖ How is the use of research or data integral to each step of the continuous 
improvement process? That is, how is each step designed so that it must be 
successfully completed by incorporating research or data? 

❖ How does your model reinforce a continuous (cyclical) process? 


Continuous 

Improvement Steps 

Your Improvement Process Steps 

Gaps or Possible Changes 

Inform: Analyze local 
needs; adjust focus. 



Select: Identify, 
examine, and select 
evidence-based 
interventions. 



Plan: Develop 
implementation 
strategies, adjust for 
local context. 



Implement: Proceed with 
improvement 
interventions; make 
formative adjustments. 



Analyze: Conduct 
summative assessment of 
performance and 
effectiveness. 
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STEP 4: Identify Current Supports 


In the table below, identify the supports that your SEA offers (e.g., webinar, guidebook, 
training, side-by-side coaching, regional forums) to build LEA capacity in each of the five 
steps of a continuous improvement process. List the supports in chronological or sequential 
order and indicate which of the steps each support relates to. In the last column, indicate 
whether the resource supports or incorporates the use of evidence or data to inform 
decisions. After filling in the table, review what you have listed and identify gaps or needed 
changes (e.g., redundancies, timing issues, lack of resources). Questions to consider: 


❖ Do the supports that you provide address all of the steps? Is the timing or sequence 
appropriate? 

❖ Are the supports strong enough to allow LEAs to navigate the process independently, 
or is additional direct support from the SEA or from intermediaries still required? 

❖ How effective are the supports, and what seems to make them effective? How do you 
know? 

❖ Does everyone who needs support receive it? 

❖ Are there adequate supports for the use of evidence or data to inform decisions? 
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Evidence? 









Gaps or Needed Changes 
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STEP 5: Identify Priorities and Next Steps 


Review the results of your inventory of current practice (steps 2-4), select priority areas to 
address, and describe next steps, including action items, persons responsible, and due 
dates. Questions to consider: 

❖ What changes in the structure of your evidence-based continuous improvement 
process need to be made in order to include each step of the recommended 
continuous improvement process, and to incorporate research or data throughout the 
process? (See the results captured in the step 3 chart.) 

❖ What can you change, add, or remove in order to have the biggest impact? 

❖ What changes in support might be most beneficial? (See the results captured in the 
step 4 chart.) 

❖ What resources (time, funding, personnel) will you need? 

❖ How will you communicate any changes to others in the SEA, to state support 
providers, and to LEAs? 


Priority Areas to Address 


Next Steps 
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Tool 2: LEA Inventory of 


Current Practice 


Purpose 


To help local education agency (LEA) staff members be explicit about the LEA’s process for 
engaging with its schools in evidence-based improvement planning and decision-making, and 
the support the LEA provides for effective use of the process; identify gaps or needed 
changes to strengthen the LEA's model or framework; and prioritize next steps. 


Outcome 


The discussions prompted by this tool will lead to a plan of action to address gaps and 
ensure the LEA's process is evidence-based and is aligned with ESSA requirements and with 
the state’s process. The completed tool will also serve as an artifact of discussions and 
decisions. 


Materials 


In preparation, gather guides, graphics, and other documentation that describe and provide 
support for your district’s processes for improvement planning, monitoring, decision-making, 
and related activities. 

Who Should Be Involved 

Staff who oversee the district’s improvement efforts should be involved in working through 
this tool. Those involved should have substantive knowledge of the relevant federal programs 
and requirements. Participants may include multiple staff from one program or department, 
or, ideally, staff from multiple programs and offices, in order to support alignment across the 
LEA. 


Time 


Set aside 1-2 hours for each step, or longer (3 or more hours) if multiple programs or 
departments are working together. This time can be spaced out over several weeks, with 
breaks between steps to reflect or to gather additional information to inform the subsequent 
step(s). 
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Instructions Overview (detailed instructions are provided with each step) 


❖ Step 1: Get organized, including identifying roles and responsibilities, gathering 
materials, establishing a schedule, and reviewing documentation. 

❖ Step 2: Describe the context of your LEA’s process for engaging with schools in 
evidence-based improvement planning and decision-making, and for providing 
support to the schools in this process. 

❖ Step 3: Map the steps of your improvement process to the steps of the recommended 
process (described in section 2 of this guide) and identify any gaps or needed 
changes in order for your district’s process to incorporate the elements of the 
recommended process. 

❖ Step 4: Identify the supports offered to schools (e.g., webinar, guidebook, training, 
side-by-side coaching, regional forums); the timing or sequence (e.g., specific dates 
or times of the year, such as fall or spring); and any gaps or needed changes (e.g., 
redundancies, timing issues, lack of resources) in order for your district’s process to 
incorporate the elements of the recommended process. 

❖ Step 5: Review your inventory of current practice (steps 2-4), select priority areas to 
address, and describe next steps. Develop an action plan. 

Leading the Conversation 

❖ One outcome of this conversation is to build a common understanding of an 
evidence-based framework that will drive improvement work. Especially in steps 2, 3, 
and 4, be aware that participants will likely have different backgrounds and 
experiences through which they view the framework and how it is implemented. 

While participants may seem to agree, it will be important to regularly check for 
understanding. Ask participants to repeat what they heard. Paraphrase contributions 
to ensure that your understanding is accurate. Encourage participant questions. 

❖ For step 3, consider having copies of the continuous improvement framework from 
section 2 of this guide, including the descriptions of each element of the process, 
available. Being able to compare the district’s recommended framework with this 
guide’s framework will inform the conversation about alignment and possible gaps. 

Modifications or Variations 

❖ For step 3, consider using this tool in two phases. In the first phase, focus exclusively 
on your LEA framework for improvement planning and on identifying strengths and 
areas for needed adjustments. In the second phase, revisit your framework and delve 
deeper into how the use of evidence and data is infused throughout. Is this use of 
evidence an explicit activity such as requiring certain data in a needs assessment or 
requiring an evidence base for interventions? Are relevant data readily available? Are 
there adequate supports built in? You might consider modifying the provided table by 
adding a column to capture details about the use of evidence or data. 
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STEP 1 : Get Organized 


Take time to get organized and build your knowledge base by gathering necessary materials 
and building a well-informed team to complete subsequent steps. Things to consider: 

Who is involved? Identify which departments or programs should be represented; then 
identify the individuals who will participate in this work. What are their respective roles and 
responsibilities in the group (e.g., note taker, facilitator)? 



What do we need to know? Gather and organize documents and other materials that 
describe the planning and related support processes to schools in your district. Consider the 
value of having graphics or models from sources other than your own department or 
program. Review documentation and ensure that all participants have strong foundational 
knowledge of the information. Does everyone have a common understanding of the subject 
matter (e.g., SEA and LEA improvement frameworks, ESSA requirements, evidence-based 
interventions)? 



Section 4: Tools | 33 










Attend to logistics. What will the meeting schedule be? Where will you meet? What 
materials do you need (e.g., chart paper, sticky notes)? 


Notes 
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STEP 2: Describe Context 


Briefly describe the context of your process for engaging with schools in evidence-based 
improvement planning and decision-making, and for providing support to schools. Questions 
to consider: 

❖ What model or framework have you defined that guides this work with schools? 

❖ What services, resources, or other supports are you required to provide to schools 
(e.g., based on funding or other programmatic requirements)? 

❖ How is evidence-based decision-making part of this process? Where does it fit in? 

❖ What is produced as a result of going through the improvement planning process? 


Briefly Describe Your Improvement Planning and Decision-making Processes 
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STEP 3: Map Current Process 


The first column of the table below outlines the five steps of a recommended evidence-based 
improvement process, including key decision-making elements (full descriptions of each step 
are provided in section 2 of this guide). In the second column, map the steps in your district's 
improvement process for schools to steps in the recommended process. Be sure to include 
how each step uses evidence. Note that there may be more than one step in your process for 
each step in the recommended process (e.g., the Inform step of the recommended process 
might include both “Needs Assessment” and “Goal Development” from your district’s 
process). In the third column, identify any gaps or needed changes in order for your district’s 
process to incorporate the elements of the recommended process. Questions to consider: 

❖ Does your district’s process address all of the steps in the recommended process? If 
not, what are the gaps? 

❖ Does your district’s process address all of the steps in your state’s process, if 
applicable? If not, what are the gaps? 

❖ How is the use of research or data integral to each step of the continuous 
improvement process? That is, how is each step designed so that it must be 
successfully completed by incorporating research or data? 

❖ How does your model reinforce a continuous (cyclical) process? 


Continuous Improvement Steps I Your Improvement Process Steps I Gaps or Possible Changes 


Inform: Analyze local needs; 
adjust focus. 


Select: Identify, examine, and 
select evidence-based 
interventions. 


Plan: Develop implementation 
strategies, adjust for local 
context. 


Implement: Proceed with 
improvement interventions; make 
formative adjustments. 


Analyze: Conduct summative 
assessment of performance and 
effectiveness. 
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STEP 4: Identify Current Supports 


In the table below, identify the supports that your LEA offers (e.g., webinar, guidebook, 
training, side-by-side coaching, regional forums) to build school capacity in each of the five 
steps of a continuous improvement process. List the supports in chronological or sequential 
order and indicate which of the steps each support relates to. Include any supports provided 
by the state or by intermediary organizations. In the last column, indicate whether the 
resource supports or incorporates the use of evidence or data to inform decisions. After filling 
in the table, review what you have listed and identify gaps or needed changes (e.g., 
redundancies, timing issues, lack of resources). Questions to consider: 

❖ Do the supports that you provide address all of the steps? Is the timing or sequence 
appropriate? 

❖ How do your supports leverage state supports? 

❖ Are the supports strong enough to allow schools to navigate the process 
independently, or is additional direct support from the district or from intermediaries 
still required? 

❖ How effective are the supports, and what seems to make them effective? How do you 
know? 

❖ Does everyone who needs support receive it? 

❖ Are there adequate supports for the use of evidence or data to inform decisions? 
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Gaps or Needed Changes 
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STEP 5: Identify Priorities and Next Steps 


Review the results of your inventory of current practice (steps 2-4), select priority areas to 
address, and describe next steps, including action items, persons responsible, and due 
dates. Questions to consider: 

❖ What changes in the structure of your evidence-based continuous improvement 
process need to be made in order to include each step of the recommended 
continuous improvement process (or, alternatively, align with the state’s process), 
and to incorporate research or data throughout the process? (See the results capture 
in the step 3 chart.) 

❖ What can you change, add, or remove in order to have the biggest impact? 

❖ What changes in support might be most beneficial? (See the step 4 chart.) 

❖ What resources (time, funding, personnel) will you need? 

❖ How will you communicate any changes to others in the district, to support providers 
(e.g., intermediaries, state support providers), and to schools? 



Section 4: Tools | 39 



©Tool 3: SEA Guidance for 
Evidence-Based Interventions 


Purpose 

To help state education agencies (SEAs) define its approach to guiding local education 
agencies (LEAs) in selecting evidence-based interventions. 

Outcome 

Completing this tool will result in a clearly defined and well-delineated approach to working 
with LEAs to select evidence-based interventions. 

Materials 

In preparation, gather ESSA requirements and resources that help interpret ESSA 
requirements that are relevant to your program or department (see section 5 of this guide for 
suggested resources); relevant policy, guidance, and feedback on your SEA’s role in school 
and district improvement decision-making; and information on capacity and staff availability. 

Who Should Be Involved 

SEA staff who oversee school and district support efforts, and those who administer or 
oversee the interventions that are relevant to your program/department, should be involved in 
working through this tool. Those involved should have substantive knowledge of SEA support 
efforts and requirements. Participants may include staff from one program or department, or, 
ideally, staff from multiple programs and offices, to support alignment across the agency. 

SEA research and evaluation staff would also help inform the approach to supporting schools 
and districts in selecting evidence-based interventions. It would be difficult to complete this 
tool without the input of individuals with strong quantitative research expertise, particularly 
with regard to how an SEA’s research capacity may influence the SEA’s approach. 

Time 

Set aside 1-2 hours for each step, or longer (3 or more hours) if multiple programs or 
departments are working together. This time can be spaced out over several weeks, with 
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breaks between steps to reflect or to gather additional information to inform the subsequent 
step(s). 

Instructions Overview (detailed instructions are provided with each step) 

❖ Step 1 : Prepare for engaging in this activity with a cross-functional SEA team by 
organizing the team and materials. 

❖ Step 2: Review the ESSA requirements for your specific program or funding stream, 
and summarize the main points. 

❖ Step 3: Reflect on the role and capacity of your SEA in guiding LEAs to select 
evidence-based interventions. 

❖ Step 4: Using the information that you summarized in previous steps, and some 
additional considerations, define your SEA's approach to guiding LEAs in selecting 
evidence-based interventions. 

❖ Step 5: Begin planning by reviewing the results of your discussions and reflections, 
selecting implementation priorities, and describing next steps. 

Leading the Conversation 

❖ This tool is designed to help an SEA define only the guidance and support offered to 
LEAs in selecting evidence-based interventions - not guidance and support in how to 
implement and monitor those interventions. Step 2 is a good time to clarify this 
purpose, so that participants understand the scope of the conversation. Team 
members may become frustrated if they expect to do more but do not have sufficient 
time or prompts to accomplish these additional tasks. 

❖ Especially if a more prescriptive approach to selecting interventions is chosen, it will 
be critical to communicate with LEAs early and often, so that they understand how 
and why decisions have been made. As part of steps 4 and 5, carefully consider what 
needs to be communicated, and when and how that information is best shared. 

Modifications or Variations 

❖ It may not be practical or appropriate for all team members to participate in each step 
or in all parts of a step. In step 1, for example, the organizers or facilitators might 
determine who will be involved and develop the meeting schedule, but the entire 
team might work together on gathering and reviewing documentation. Organizers and 
facilitators will have to determine what works best for their participants in their setting. 

❖ Some SEA departments may choose different approaches than other departments, 
due to variations in ESSA requirements for their funding streams. If so, consider 
bringing the departments together, after completing this tool, to describe the overall 
SEA approach to supporting LEAs in selecting evidence-based interventions, and to 
determine how the overall and department-specific approaches will be communicated 
to LEAs. 
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STEP 1 : Get Organized 


Take time to get organized and build your knowledge base by gathering necessary materials 
and building a well-informed team to complete subsequent steps. Things to consider: 

Who is involved? Identify which departments or programs should be represented; then 
identify the individuals who will participate in this work. What are their respective roles and 
responsibilities in the group (e.g., note taker, facilitator)? 


Department or 

Program 

Name 

Contact Information 

Role/Responsibility 

















\~~ ~~ 

\~— ~~ 

\~— ~~ 



What do we need to know? Gather and organize documents and other materials related to 
your SEA’s current approach to supporting LEAs in selecting evidence-based interventions. 
Review policy guidance, feedback reports, or evaluations on the role and capacity of your 
SEA and/or department. 


Document or Material 
(title, description, source) 

Why It Is Important 

(what everyone should know) 
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Attend to logistics. What will the meeting schedule be? Where will you meet? What 
materials do you need (e.g., chart paper, sticky notes)? 


Notes 
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STEP 2: Review ESSA Requirements 


ESSA calls on SEAs to take an increased role in supporting, managing, and monitoring the 
implementation of evidence-based decision-making at the local level. This tool is focused on 
defining an SEA approach to guiding LEA selection of evidence-based interventions. This 
includes defining the level of specificity and prescriptiveness for how LEAs select these 
interventions. For example, the SEA might provide a specific list of interventions from which 
LEAs must choose, or develop a process for approving LEAs' decisions, or support an open- 
ended process to review LEAs' choices. For additional information related to ESSA 
provisions, see section 2, and the resources listed in section 5, of this guide. 

As you review the ESSA requirements for your specific program or funding stream, consider 
the following questions: 

❖ What element of your program or funding stream requires evidence-based 
interventions? 

❖ What are the situations (e.g., high schools with low graduation rates) that trigger 
evidence-based interventions? 

❖ What level of evidence is required for interventions in your program or funding 
stream? 

❖ What is required in terms of SEA review and approval of plans for your program or 
funding stream? What are the implications for your SEA's role and capacity? 


Notes: Review and Summarize the ESSA Requirements 
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STEP 3: Reflect on Your SEA’s Role and Capacity to 
Engage with LEAs 


What skills or knowledge do your SEA staff possess to guide LEAs in the process of selecting 
evidence-based interventions? What does your team need to consider about roles and 
capacity before defining an approach for selecting evidence-based interventions? Note that 
capacity encompasses a number of dimensions, from human capacity (Do you have the 
people to do it?) to organizational capacity (Do you have the processes and structures to 
guide this work?) and material capacity (Do you have the technology and necessary 
materials to support the work?). The following questions are intended to help guide your 
conversations with your SEA colleagues to determine your approach. 

SEA Role 

❖ How does our SEA's past practice or policy regarding the degree of centralized or 
local control impact how we choose to work with LEAs to select evidence-based 
interventions? 

❖ How prescriptive do we want to be as a state agency? 

❖ What authority does our SEA have under state laws, policies, and regulations? 

❖ What have our past approaches to providing guidance to LEAs in other programs or 
funding streams been? What would we like to keep or change about how we 
approach providing guidance to LEAs? What changes do we need to make to align 
with ESSA requirements? 

SEA Capacity 

❖ What is our SEA's capacity to conduct or review and summarize research on the 
effectiveness of interventions? What resources outside of the SEA, such as our 
Regional Educational Laboratory, can assist us? 

❖ What is our SEA's capacity to approve LEAs’ approaches? 

❖ What is our SEA's capacity to provide technical assistance to LEAs on selecting 
evidence-based interventions? 

❖ What is our SEA's capacity to monitor LEAs’ evidence-based intervention selection 
processes? 
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Summarize and Reflect on the Role and Capacity of Your SEA. 
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STEP 4: Define Your SEA’s Approach 


Considering the role and capacity of your SEA in selecting evidence-based interventions, 
what will your approach be under ESSA? Consider the following questions: 

Clarify Your Approach 

❖ Under the Elementary and Secondary Education Act/No Child Left Behind, what was 
our state's approach to guidance on selecting evidence-based interventions? What 
lessons did we learn about that approach that can be applied to our ESSA approach? 

❖ How does our specific program or funding stream shape our approach? 

❖ How does the timeline for planning or implementation within our specific program or 
funding stream affect our approach? 

❖ Can additional information from our summaries in previous steps of this tool be used 
to influence how we will design our approach? 

❖ Under what conditions would we differentiate our approach for LEAs? 

Clarify Your Support 

❖ What kinds of support will our SEA provide to help LEAs make local decisions around 
evidence-based interventions? 

❖ How will we describe our approach to LEAs? How will we describe it to other 
departments within the SEA? 

❖ What will our SEA produce to guide LEAs? A list of interventions to choose from, a 
guidance document that shares the criteria that the SEA will use to vet LEA plans, or 
something else? 

❖ What resources — staffing, funding, and training — will be needed for following through 
on and sustaining our proposed approach? 

❖ What are our immediate next steps? 
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Summarize your SEA’s approach and the support that you will offer to LEAs in selecting 
evidence-based interventions. 
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STEP 5: Plan Next Steps 


Review the results of your discussions and reflections, select implementation priorities, and 
describe next steps. 


Implementation Priorities 


Next Steps 
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©Tool 4: LEA Guidance for 
Evidence-Based Interventions 


Purpose 

To help a local education agency (LEA) define its approach to guiding schools in selecting 
evidence-based interventions. 

Outcome 

Completing this tool will result in a clearly defined and well-delineated approach to working 
with schools to select evidence-based interventions. 

Materials 

In preparation, gather ESSA requirements and resources that help interpret ESSA 
requirements that are relevant to your program or department (see section 5 of this guide for 
some suggested resources); SEA guidance on evidence-based interventions; relevant policy, 
guidance, and feedback on your LEA's role in school improvement decision-making; and 
information on capacity and staffing availability. 

Who Should Be Involved 

LEA staff who oversee school support efforts and interventions should be involved in working 
through this tool. Those involved should have substantive knowledge of LEA support efforts 
and requirements. Participants may include multiple staff from one program or department, 
or, ideally, staff from multiple programs and offices, to support alignment across the LEA. If 
available, LEA research and evaluation staff with strong quantitative skills would also help 
inform the approach to selecting evidence-based interventions. 

Time 

Set aside 1-2 hours for each step, or longer (e.g., 3 or more hours) if multiple programs or 
departments are working together. This time can be spaced over several weeks, with breaks 
between steps to reflect or to gather additional information to inform subsequent steps. 
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Instructions Overview (detailed instructions are provided with each step) 


❖ Step 1 : Prepare for engaging in this activity with a cross-functional LEA team by 
organizing the team and materials. 

❖ Step 2: Review the ESSA requirements for your specific program or funding stream, 
and summarize the main points. 

❖ Step 3: Reflect on the role and capacity of your LEA in guiding schools to select 
evidence-based interventions. 

❖ Step 4: Using the information that you summarized in previous steps, and some 
additional considerations, define your LEA's approach to guiding schools in selecting 
evidence-based interventions. 

❖ Step 5: Begin planning by reviewing the results of your discussions and reflections, 
selecting implementation priorities, and describing next steps. 

Leading the Conversation 

❖ This tool is designed to help an LEA define only the guidance and support offered to 
schools in selecting evidence-based interventions — not guidance and support in how 
to implement and monitor those interventions. Step 2 is a good time to clarify this 
purpose, so that participants understand the scope of the conversation. Team 
members may become frustrated if they expect to do more but do not have sufficient 
time or prompts to accomplish these additional tasks. 

❖ Especially if a more prescriptive approach to selecting interventions is chosen, it will 
be critical to communicate with schools early and often, so that they understand how 
and why decisions have been made. As part of steps 4 and 5, carefully consider what 
needs to be communicated, and when and how that information is best shared. 

Modifications or Variations 

❖ Depending on needs, capacity, and logistical considerations, this tool can be 
completed by an LEA independently or with SEA support. 

❖ It may not be practical or appropriate for all team members to participate in each step 
or in all parts of a step. In step 1, for example, the organizers or facilitators might 
determine who will be involved and develop the meeting schedule, but the entire 
team might work together on gathering and reviewing documentation. Organizers and 
facilitators will have to determine what works best for their participants in their setting. 

❖ Some LEA departments may choose different approaches than other departments, 
due to variations in ESSA requirements for their funding streams. If so, consider 
bringing departments together, after completing this tool, to describe the overall LEA 
approach to supporting schools in selecting evidence-based interventions, and to 
determine how the overall and department-specific approaches will be communicated 
to schools. 
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STEP 1 : Get Organized 


Take time to get organized and build your knowledge base by gathering necessary materials 
and building a well-informed team to complete subsequent steps. Things to consider: 

Who is involved? Identify which departments or programs should be represented; then 
identify the individuals who will participate in this work. What are their respective roles and 
responsibilities in the group (e.g., note taker, facilitator)? 


Department 
or Program 

Name 

Contact Information 

Ro le/Responsi bi lity 


















\~— ~~ 

\~— ~~ 



What do we need to know? Gather and organize documents and other materials related to 
your LEA's current approach to supporting schools in selecting evidence-based interventions. 
Review policy guidance, feedback reports, or evaluations on the role and capacity of your 
LEA and/or department. 


Document or Material 
(title, description, source) 

Why It Is Important 

(what everyone should know) 
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Attend to logistics. What will the meeting schedule be? Where will you meet? What 
materials do you need (e.g., chart paper, sticky notes)? 
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STEP 2: Review ESSA Requirements 


ESSA calls on LEAs to take an increased role in supporting, managing, and monitoring the 
implementation of evidence-based decision-making at the local level. This tool is focused on 
defining an LEA approach to guiding schools’ selection of evidence-based interventions. 
Under SEA guidance, and in alignment with SEA approaches, LEAs can define the level of 
specificity and prescriptiveness for how schools select these interventions. For example, the 
LEA might provide a specific list of interventions from which schools must choose, or develop 
a process for approving schools’ decisions, or support an open-ended process to review 
schools’ choices. For additional information related to ESSA provisions, see section 2, and 
the resources listed in section 5, of this guide. 

As you review the ESSA requirements for your specific program or funding stream, consider 
the following questions: 

❖ What element of your program or funding stream requires evidence-based 
interventions? 

❖ What are the situations (e.g., high schools with low graduation rates) that trigger 
evidence-based interventions? 

❖ What level of evidence is required for interventions in your program or funding 
stream? 

❖ What is required in terms of SEA review and approval of plans for your program or 
funding stream? What are the implications for your LEA's role and capacity? 


Review and Summarize the ESSA Requirements and SEA Guidance 
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STEP 3: Reflect on Your LEA’s Role and Capacity to 
Engage with Schools 


What skills or knowledge do your LEA staff possess to guide schools, with SEA support, in 
the process of selecting evidence-based interventions? What does your team need to 
consider about roles and capacity before defining an approach for selecting evidence-based 
interventions? Note that capacity encompasses a number of dimensions, from human 
capacity (Do you have the people to do it?) to organizational capacity (Do you have the 
processes and structures to guide this work?) and material capacity (Do you have the 
technology and necessary materials to support the work?). The following questions are 
intended to help guide your conversation with your SEA and LEA colleagues to determine 
your approach. 

LEA Role 

❖ How does our LEA's past practice or policy regarding the degree of centralized or 
local control impact how we choose to work with schools to select evidence-based 
interventions? 

❖ How prescriptive do we want to be as a district? How prescriptive is our SEA? 

❖ What authority does our LEA have under state laws, policies, and regulations? 

❖ What have our past approaches to providing guidance to schools been? What would 
we like to keep or change about how we approach providing guidance to schools? 
What changes do we need to make to align with ESSA requirements? 

LEA Capacity 

❖ What is our LEA's capacity to conduct or review research on, and to assess 
evidence-based interventions? What resources outside of the LEA (e.g., SEA, 
Regional Educational Laboratory, county office of education, outside organizations) 
can support these efforts? 

❖ What is our LEA's capacity to approve schools’ approaches and to determine how 
those approaches fit within SEA guidelines? 

❖ What is our LEA's capacity to provide technical assistance to schools on selecting 
evidence-based interventions? How is our capacity influenced by SEA processes? 

❖ What is our LEA's capacity to monitor schools’ evidence-based intervention selection 
processes? How is our capacity influenced by SEA processes? 
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Summarize Your Reflection on the Role and Capacity of Your LEA. 
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STEP 4: Define Your LEA’s Approach 


Considering the role and capacity of your LEA in selecting evidence-based interventions, 
what will your approach be under ESSA? Consider the following questions: 

Clarify Your Approach 

❖ Under the Elementary and Secondary Education Act/No Child Left Behind, what was 
our approach to guidance on selecting research-based or evidence-based 
interventions? What lessons did we learn about that approach that can be applied to 
our ESSA approach? 

❖ How does our specific program or funding stream shape our approach? 

❖ How does the timeline for planning or implementation within our specific program or 
funding stream affect our approach? 

❖ Can additional information from our summaries in previous steps of this tool be used 
to influence how we will design our approach? 

❖ How does the SEA approach influence our LEA approach? 

❖ Under what conditions would we differentiate our approach for schools? 

Clarify your support 

❖ What kinds of support will our LEA provide to help schools make local decisions 
around evidence-based interventions? 

❖ How will we describe our approach to schools? How will we describe it to other 
departments within the LEA? How will we describe it to the SEA? 

❖ What will our LEA produce to guide schools? A list of interventions to choose from, a 
guidance document that shares the criteria that the LEA will use to vet schools' plans, 
or something else? How will SEA guidance and/or materials shape our guidance 
materials? 

❖ What kinds of support will our LEA provide to help schools make local decisions 
around evidence-based interventions? How does this support reflect SEA support 
and guidance? 

❖ What resources — staffing, funding, and training — will be needed for following through 
on and sustaining our proposed approach? 

❖ What are our immediate next steps? 
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Summarize Your LEA’s Approach to Guiding Schools in Selecting Evidence-based 
Interventions. 
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STEP 5: Plan Next Steps 


Review the results of your discussions and reflections, select implementation priorities, and 
describe next steps. 


Implementation Priorities 


Next Steps 
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O^Tool 5: Intervention Evidence 
Review 


Purpose 

To help a state education agency (SEA), local education agency (LEA), or school review 
research on interventions, strategies, policies, practices, programs, or activities (collectively 
referred to, in this tool, as interventions) that target a specific issue, problem, or outcome. 

Outcome 

By completing this tool, the user can assess the entire body of evidence (based on the 
evidence definitions in ESSA and nonregulatory guidance from the U.S. Department of 
Education [ED]) for particular interventions that target the outcome of interest. 

Materials 

Gather results from completed needs assessments, or root-cause or gap analyses; 
statements or summaries of problems or issues that require attention; lists of interventions 
that you currently use; and research on potential interventions. Also, gather any relevant 
information about the setting of your state, district, or school (e.g., populations served). 

Who Should Be Involved 

Staff who are charged with selecting evidence-based interventions for states, districts, or 
schools should work together to use this tool. Those involved should have substantive 
knowledge of the issues and outcomes that potential interventions are intended to address, 
including the educational setting(s) where the intervention would be implemented (e.g., 
knowledge of the targeted grade levels and student populations). Staff, consultants, or 
providers of technical assistance (e.g., staff from your Regional Educational Laboratory) who 
have a background in quantitative research methods should assist with the review of the 
research literature and completion of the tool. 

Time 

Completing the first two questions and the last two rows of the tool — tasks in which the entire 
team is engaged — requires approximately one hour. In addition, 45 to 90 minutes per study, 
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depending on the length and complexity of the study, are required for reviewing each 
research study. 

Instructions Overview (detailed instructions are provided with each step) 

❖ Step 1: Get organized, including identifying roles and responsibilities, gathering 
materials, establishing a schedule, and reviewing documentation. 

❖ Step 2: Take stock of the most pressing problems or issues in your education setting, 
the outcomes that you would like to achieve, and possible interventions to help 
achieve those outcomes. 

❖ Step 3: Review available research studies on a single intervention and determine 
whether each study demonstrates strong, moderate, or promising evidence for the 
intervention (or demonstrates a rationale for using the intervention), based on ESSA 
standards and nonregulatory guidance from ED. 

❖ Step 4: Examine the context of the research and determine the extent to which the 
evidence for each study on the intervention was gathered in educational settings and 
populations similar to yours. 

❖ Step 5: Assess the cumulative body of evidence on the intervention, both in general 
and specifically for your educational setting. 

Leading the Conversation 

❖ The goal of this exercise is to examine the entire body of research that exists for an 
intervention. Although ESSA stipulates that a single study may be used to provide 
strong, moderate, or promising evidence, subsequent nonregulatory guidance from 
ED encourages stakeholders to examine the body of research on an intervention. 

❖ The facilitator of the activity might want to begin the activity by clarifying differences 
in how evidence is used in decision-making in ESSA, compared to No Child Left 
Behind to help team members understand the importance of their work. See section 
2 of this guide for an explanation of the role of research and evidence in decision- 
making in ESSA and for a graphic showing a recommended evidence-based 
decision-making process. 

Modifications or Variations 

❖ It may not be practical or appropriate for all team members to participate in each step 
or in all parts of a step. In step 1, for example, the organizers or facilitators might 
determine who will be involved and develop the meeting schedule, but the entire 
team might work together on gathering and reviewing documentation. Organizers and 
facilitators will have to determine what works best for their participants in their setting. 

❖ Multiple experts in quantitative research methods can be called upon to find the 
relevant research and to review the research. If multiple experts are used, they 
should first complete the tool independently and then meet as a group to discuss and 
compare results and to note discrepancies among their results. 
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❖ This tool can be used in at least two ways. It is designed to help compare the 
evidence bases for more than one intervention, to identify what has the best 
likelihood of addressing a well-defined problem or outcome. In certain cases, there 
may be a large number of interventions that target the outcome of interest, whereas, 
in other cases, there may be fewer such interventions. In situations where there are 
many interventions, users of this tool may want to first group the interventions into 
different categories (e.g., classroom-focused versus whole-school, or delivered by an 
instructor versus delivered online) in order to facilitate decision-making. The tool can 
also be used to understand what research shows about the success of a single 
intervention in different contexts, so that the intervention can be intentionally adapted 
to local context. 
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STEP 1 : Get Organized 


Take time to get organized and build your knowledge base by gathering necessary materials 
and building a well-informed team to complete subsequent steps. Things to consider: 

Who is involved? Identify which departments or programs should be represented; then 
identify the individuals who will participate in this work. What are their respective roles and 
responsibilities in the group (e.g., note taker, facilitator)? The team should include individuals 
who have input into selecting the evidence-based interventions; stakeholders who have 
knowledge of the problems, issues, and context of the educational setting; and one or more 
experts in quantitative research methods. 


Department 
or Program 

Name 

Contact Information 

Ro le/Responsi bi lity 




1 




1 




1 




1 


\~— ~~ 

\~~ ~~ 

J 


What do we need to know? Gather and organize documents and other materials that 
provide data about or describe the most pressing problems or issues in your state, district, or 
school. These materials could include results from completed needs assessments or gap 
analyses and from logic models, as well as graphics that depict problems or issues along with 
their effects and possible causes. You may also want to develop a list of interventions that 
are currently in place at the state, district, and school levels. 


Document or Material 
(title, description, source) 

Why It Is Important 

(what everyone should know) 










1 
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Attend to logistics. What will the meeting schedule be? Where will you meet? What 
materials do you need (e.g., chart paper, sticky notes)? Who will be involved in step 3 
(reviewing the research)? 


Notes 
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STEP 2: Take Stock of Problems or Issues 


Convene the entire team, review the materials, and decide what the most pressing problems, 
issues, or questions in your education setting are, in relation to the program or funding 
stream under consideration; the outcomes that you would like to achieve; and possible 
interventions to help achieve those outcomes. 


Question 1. Think about your educational setting (state, district, or school). Based on a 
needs assessment or other analysis, what are the two or three most pressing problems, 
issues, or questions that you would like addressed in relation to the program or funding 
stream under consideration? 


Question 2. Based on the pressing problems, issues, or questions that you would like 
addressed (e.g., improving the high school graduation rate), what specific outcome(s) 
are you hoping to achieve, and for whom? 
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Question 3. What are some of the interventions currently in place in your state or 
district, and/or that you might consider for implementation, that focus on improving the 
specific outcome(s) that you are hoping to change for your population? 


Select one of the interventions that attempts to address the problem or issue and to 
achieve the targeted outcomes, and write the intervention in the space below. Carry 
over the intervention to Step 3 (Repeat this step for each intervention under 
consideration.) 
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STEP 3: Review the Research 


Before beginning this step, the team should consult the What Works Clearinghouse (WWC), 
which contains extensive information on a number of education-focused interventions. If there 
are no available evidence reviews on the intervention, primary research on the intervention 
can be obtained from academic databases such as the Education Resources Information 
Center (ERIC). Consult section 5 for other sources of evidence reviews of social programs, 
including those that are focused on education. 

The researcher(s) on the team should obtain and review all of the available research on the 
intervention under study, and determine whether each study provides strong, moderate, or 
promising evidence for the intervention (or demonstrates a rationale for adoption), based on 
ESSA standards and guidance from ED, as well as on the nonregulatory guidance that 
applies to evidence standards in education. The ESSA standards and guidance from ED will 
also provide definitions of randomized controlled trials, quasi-experimental design, and 
correlational study. These resources should also be consulted for definitions of “well- 
implemented” and “well-designed” research. If more than one researcher is completing these 
questions, they should consult each other regarding their responses and resolve any 
discrepancies. 


Transfer the intervention you selected at the end of the previous step here. 


Question 

In each column, address the question 
for each study, using the response 
options listed; make notes in the 
columns if needed. 

Study 1 

Study 2 

Study 3 

4 

List the source of the evidence review 
(e.g., WWC), or author(s) and year of 
the study 




5a 

Was this study a well-designed and well- 
implemented randomized controlled 
trial (RCT), as defined by ESSA? 

Response options: Yes, No, or Not enough 
information 

If “Yes,” go to 5b. 

If “No” or “Not enough information,” go 
to 6a. 
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Question 



5b 


5c 


5d 


6b 


In each column, address the question 
for each study, using the response 
options listed; make notes in the 
columns if needed. 

Study 1 

Study 2 

Study 3 

For this RCT, is there a statistically 
significant favorable effect of the 
intervention on the relevant 
outcome(s)? 

Response options: Yes or No 




For this RCT, is there a statistically 
significant and overriding unfavorable 
effect on the relevant outcome(s)? 

Response options: Yes or No 




Does this RCT provide STRONG EVIDENCE 
for the intervention? 

Response options: Yes or No 

If the response to row 5b is “Yes” and the 
response to 5c is “No,” mark “Yes” in this 
row, then proceed to question 9. 

If the response to row 5b is “No” and the 
response to row 5c is “Yes,” mark “No” in 
this row, then proceed to question 6a. 




Was this study a well-designed and well- 
implemented quasi-experimental design 
(QED), as defined by ESSA? 

Response options: Yes, No, or Not enough 
information 

If “Yes,” go to 6b. 

If “No” or “Not enough information,” go 
to 7a. 




For this QED study, is there a 
statistically significant favorable impact 
of the intervention on the relevant 
outcome(s)? 

Response options: Yes or No 
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Question 


In each column, address the question Study 1 Study 2 

for each study, using the response 
options listed; make notes in the 
columns if needed. 


6c For this QED study, is there a 

statistically significant and overriding 
unfavorable effect on the relevant 
outcome(s)? 

Response options: Yes or No 


6d Does this QED study provide MODERATE 
EVIDENCE for the intervention? 

Response options: Yes or No 

If the response to row 6b is “Yes” and the 
response to 6c is “No,” mark “Yes” in this 
row, then proceed to question 9. 

If the response to row 6b is “No” and the 
response to row 6c is “Yes,” mark “No” in 
this row, then proceed to question 7a. 


7a Was this study a well-designed and well- 
implemented correlational study, as 
defined by ESSA? 

Response options: Yes, No, or Not enough 
information 

If “Yes,” go to 7b. 

If “No” or “Not enough information,” go 
to 8. 

7b For this correlational study, is there a 
statistically significant favorable impact 
of the intervention on the relevant 
outcome(s)? 

Response options: Yes or No 


7c For this correlational study, is there a 
statistically significant and overriding 
unfavorable effect on the relevant 
outcome(s)? 

Response options: Yes or No 


Study 3 








Question 


In each column, address the question Study 1 Study 2 

for each study, using the response 
options listed; make notes in the 
columns if needed. 


7d Does this correlational study provide 
PROMISING EVIDENCE for the 
intervention? 

Response options: Yes or No 

If the response to 7b is “Yes” and the 
response to 7c is “No,” mark “Yes” in this 
row, then proceed to question 9 under 
step 4. 

If the response to 7b is “No” and the 
response to 7c is “Yes,” mark “No” in this 
row, then proceed to question 8. 

8 Does this study DEMONSTRATE A 

RATIONALE for using the intervention? In 
other words, does this study suggest that 
the intervention is likely to improve a 
relevant outcome? 

Response options: Yes, No, or Not enough 
information 


Study 3 







STEP 4: Examine the Research and Educational 
Context 


Convene the entire team to review the studies of this intervention. Then, assess the similarity 
of your educational setting, and of the population(s) of interest, to those used in the studies of 
the intervention. 


Transfer the intervention you selected at the end of step 2 here. 


Question 

In each column, address the question 
for each study, using the response 
options listed; make notes in the 
columns if needed. 

Study 1 

Study 2 

Study 3 

9 

Examining the information for questions 

5d, 6d, 7d, and 8, what is the highest 
level of evidence provided by each study 
for the intervention? 

Response options: Strong, Moderate, 
Promising, or Demonstrates a rationale 




10 

In each study, was the intervention 
implemented in a setting similar to yours 
(e.g., rural, urban, grade span)? 

Response options: Very much, Somewhat, A 
little, Not at all 



11 

In each study, was the intervention 
implemented with populations similar to 
yours (e.g., limited language proficient, 
low socio-economic status, specific 
ethnicity)? 

Response options: Very much, Somewhat, A 
little, Not at all 
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STEP 5: Assess the Cumulative Evidence for the 
Intervention 


Based on the responses to previous questions in this tool, the entire team can assess the 
cumulative evidence for the intervention, both in general and specifically for your particular 
educational setting. This information can be used when completing tool 6 (Comparing 
Evidence-Based Interventions). 


12 Looking across the information for question 9 only, which of the following designations best 
describes the cumulative evidence for this intervention? 

Strong Moderate Promising Demonstrates a rationale 


13 Looking across information for questions 9, 10, and 11, which of the following designations 
best describes the cumulative evidence for this intervention, specifically for the settings 
and populations that are similar to yours? 

Strong Moderate Promising Demonstrates a rationale 
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© Tool 6: Comparing Evidence- 
Based Interventions 


Purpose 

This tool is to be used to compare how well different evidence-based interventions, 
strategies, policies, practices, or programs (collectively referred to, in this tool, as 
interventions) that target the same problem or issue are aligned with the context of a specific 
state, district, or school. 

This tool can be used after completing tool 5, or without completing tool 5 if users 
already have sufficient information about the levels of evidence for specific 
interventions under consideration for selection. 

Outcome 

Completing the tool provides an indication of the degree to which a given intervention aligns 
with the state’s, district’s, or school’s specific context. Reviewing this information will help in 
the selection of an evidence-based intervention. 

Materials 

Gather the results from tool 5 (Intervention Evidence Review), as well as information about 
each intervention’s training and implementation requirements and costs. Also, gather any 
information about the context of your educational setting (e.g., the nature of any currently 
implemented school-improvement interventions; numbers of district or school staff). 

Who Should Be Involved 

Staff who are charged with selecting evidence-based interventions for districts or schools 
should work together to use this tool. Those involved should have substantive knowledge of 
the issues and outcomes that potential interventions are intended to address, including the 
educational setting(s) where the interventions would be implemented (e.g., knowledge of the 
targeted grade levels and student populations). Staff, consultants, or technical assistance 
providers who have a background in quantitative research methods should assist with 
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completion of the tool. Ideally, those who participated in reviewing the research studies on 
the interventions (using tool 5) should participate in this discussion. 

Time 

If the user is comparing three different interventions, set aside at least 2 hours for an 
individual or a small group to complete the tool for a given intervention. Provide an additional 
3 to 4 hours for the full group to discuss and compare the different interventions, based on 
their completion of the tool. The time required for both the individual intervention reviews and 
the group discussion will increase if more than three interventions are being compared. 

Instructions Overview (detailed instructions are provided with each step) 

❖ Step 1: Get organized, including identifying roles and responsibilities, gathering 
materials, establishing a schedule, and reviewing documentation. 

❖ Step 2: Take stock of the most pressing problems or issues in your education setting, 
the outcomes that you would like to achieve, and possible interventions to help 
achieve those outcomes. 

❖ Step 3: Review available information to better understand how well each evidence- 
based intervention under consideration would fit into the context of your educational 
setting 

❖ Step 4: Review available information to determine the costs of implementing the 
interventions under consideration in your educational setting. 

❖ Step 5: Discuss the feasibility of selecting and implementing each intervention in your 
educational setting, and the advantages and disadvantages of each intervention as it 
pertains to your educational context. 

Leading the Conversation 

❖ The conversation among stakeholders using this tool should revolve around which 
evidence-based interventions best fit the contexts of their particular education 
environments. After the tool is completed, the group may not yet have decided on 
which intervention(s) to implement, but it should have narrowed down the options. 

Modifications or Variations 

❖ If your team has used tool 5, it may be possible to skip parts of steps 1 and 2. Review 
these steps carefully to see what should be repeated, reviewed, or possibly skipped. 

❖ In certain cases, there may be a large number of interventions that target the 
outcome of interest, whereas, in other cases, there may be fewer such interventions. 
In situation where there are many interventions, users of this tool may want to first 
group the interventions into different categories (e.g., classroom-focused versus 
whole-school, or delivered by an instructor versus delivered online) in order to 
facilitate decision-making. 
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❖ Questions in this tool are designed to be appropriate for a broad array of educational 
contexts. Some may be skipped or refined, or questions may be added, to fit a specific 
context. 

❖ It may not be practical or appropriate for all team members to participate in each step 
or in all parts of a step. In step 1, for example, the organizers or facilitators might 
determine who will be involved and develop the meeting schedule, but the entire 
team might work together on gathering and reviewing documentation. Organizers and 
facilitators will have to determine what works best for their participants in their setting. 
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STEP 1 : Get Organized 


Take time to get organized and build your knowledge base by gathering necessary materials 
and building a well-informed team to complete subsequent steps. Things to consider: 

Who is involved? Identify which departments or programs should be represented; then 
identify the individuals who will participate in this work. What are their respective roles and 
responsibilities in the group (e.g., note taker, facilitator)? The team should include individuals 
who have input into selecting the evidence-based interventions, and stakeholders who have 
knowledge of the problems, issues, and context of the educational setting; other interventions 
being implemented in the educational setting; the current professional development 
requirements placed on the staff in the educational setting; and any funding available for 
(where applicable) purchasing and implementing a new intervention. The team should also 
include one or more experts in quantitative research methods. 


Department 
or Program 

Name 

Contact Information 

Role/Responsibility 




~1 




~1 










What do we need to know? Gather and organize documents and other materials that 
provide data about or describe the most pressing problems or issues in your state, district, or 
school. These materials could include results from completed needs assessments or gap 
analyses and from logic models, as well as graphics that depict problems or issues along with 
their effects and possible causes. Also, include documentation that summarizes other 
interventions that are currently being implemented in the educational setting and the current 
professional development requirements placed on staff. Reviewing this documentation will 
ensure that all participants have strong foundational knowledge of the problems, desired 
outcomes, and interventions under consideration. 


Document or Material 
(title, description, source) 

Why It Is Important 

(what everyone should know) 


1 


~1 


~1 
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Attend to logistics. What will the meeting schedule be? Where will you meet? What 
materials do you need (e.g., chart paper, sticky notes)? 


Notes 
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STEP 2: Take Stock of Problems or Issues 


Convene the entire team, review the materials, and decide what the most pressing problems 
in your education setting are, in relation to the program or funding stream under 
consideration; the outcomes that you would like to achieve; and possible interventions to 
help achieve those outcomes. 


Question 1. Think about your educational setting (state, district, or school). Based on a 
needs assessment or other analysis, what are the two or three most pressing problems, 
issues, or questions that you would like addressed in relation to the program or funding 
stream under consideration? 


Question 2. Based on the pressing problems, issues, or questions that you would like 
addressed (e.g., improving school engagement among middle school students), what 
specific outcome(s) are you hoping to achieve, and for whom? 
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Question 3. What are some of the interventions currently in place in your state or 
district, and/or that you might consider for implementation, that focus on improving the 
specific outcome(s) that you are hoping to change for your population? 


Select one or more possible interventions that attempt to address the problem or issue 
and to achieve the targeted outcomes, and write the intervention(s) in the space below. 
Transfer the interventions to the table in step 3 


Section 4: Tools | 81 


Question 


STEP 3: Determine Level of Evidence and Alignment 
with Context 


Convene the entire team to assess the similarity of your educational setting, and of the 
population(s) of interest, to those used in the research on the intervention(s) selected in step 
2. Also, consider if the intervention(s) have already been adapted to your setting. If the team 
has completed tool 5, the team should refer to the completed tool as well as any other 
documentation on the interventions. 

Convene members of the team who have the most knowledge about the context of the state, 
district, or school where the intervention(s) would be adopted. Contextual factors include the 
nature of any other interventions currently being implemented in the state, district, or school, 
and staff capacity (in terms of both knowledge and time). In this step, address as many 
questions as possible that are relevant to the interventions under consideration and to your 
educational setting. You may not be able to address all of the questions with the information 
that you have at this point. The importance of the questions in this section may vary across 
states, districts, or schools. Individual questions in this section may be skipped or refined as 
needed, or questions may be added, to fit each context. 


In each column, address the 
question for each intervention, 
using the response options listed; 
make notes in the columns if 
needed. 

Intervention 

1 

Intervention 

2 

Intervention 

3 

Name of intervention 




Which of the following designations 
best describes the cumulative 
evidence for this intervention? 

(Information from tool 5 can be used to 
answer this question.) 

Response options: Strong, Moderate, 
Promising, or Demonstrates a rationale 




Which of the following designations 
best describes the cumulative 
evidence across studies for this 
intervention for settings and 
populations that are similar to yours ? 
(Information from tool 5 can be used to 
answer this question.) 

Response options: Strong, Moderate, 
Promising, or Demonstrates a rationale 
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Question 


In each column, address the 
question for each intervention, 
using the response options listed; 
make notes in the columns if 
needed. 

Intervention 

1 

Intervention 

2 

Intervention 

3 

Given the information you have, has 
the intervention already been adapted 
for your population and/or setting? 

Response options: Yes or No 

If “Yes,” go to question 9 in step 4. 

If “No,” go to question 8. 




Do you have staff or consultants who 
will be able to adapt the intervention 
while preserving the core 
components? 

Response options: Yes or No 



What existing interventions might the 
proposed intervention be in direct 
conflict with or need to be integrated 
with (e.g., in terms of staffing, 
resources, facilities, scheduling)? 



To what extent would the 
intervention fit within the known 
climate or culture of the state, 
district, or school? 

Response options: A lot, Somewhat, A 
little 



To what extent would there be 
leadership support at the district or 
school level to ensure that the 
intervention would be implemented 
with fidelity? 

Response options: A lot, Somewhat, A 
little 
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Question 


In each column, address the 
question for each intervention, 
using the response options listed; 
make notes in the columns if 
needed. 

Intervention 

1 

Intervention 

2 

Intervention 

3 

To what extent would there be buy-in 
from the necessary constituents (e.g., 
staff, community, students, and 
parents) for the intervention to be 
implemented with fidelity? 

Response options: A lot, Somewhat, A 
little 




To what extent does the state, 
district, or school have the physical 
infrastructure (e.g., space, 
technology, data systems) to fully 
implement, support, and sustain the 
intervention after funding ends? 

Response options: A lot, Somewhat, A 
little 



To what extent does the state, 
district, or school have the 
organizational structure to fully 
implement, support, and sustain the 
intervention after funding ends? 

Response options: A lot, Somewhat, A 
little 



To what extent would staff have the 
capacity and time required to 
successfully deliver and implement 
the intervention? 

Response options: A lot, Somewhat, A 
little 



To what extent would staff have the 
time required and capacity to 
continually monitor and collect data 
on implementation and outcomes of 
the intervention? 

Response options: A lot, Somewhat, A 
little 
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Question 


In each column, address the 
question for each intervention, 
using the response options listed; 
make notes in the columns if 
needed. 

Intervention 

1 

Intervention 

2 

Intervention 

3 

If applicable, will the partners or 
other outside supports that are 
necessary for implementation of the 
intervention be committed and 
available for the duration? 

Response options: Yes, No, N/A 
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STEP 4: Estimate Needed Resources 


Convene members of the team who have the most knowledge about the costs of the given 
intervention(s) as well as knowledge of the time required to implement the intervention(s) — 
both in the start-up phase and in the long term, including requirements for staff training. In 
this step, address as many questions as possible that are relevant to the intervention(s) 
under consideration and to your educational setting. You may not be able to address all of 
the questions with the information that you have at this point. The importance of the 
questions in this section may vary across states, districts, or schools. Individual questions in 
this section may be skipped or refined as needed, or questions may be added, to fit each 
context. 


Question 

In each column, address the 
question for each intervention, 
make notes in the columns if 
needed. 

Intervention 1 

Intervention 2 

Intervention 3 

18 

How much time would be 
required for staff training, in 
terms of hours or days? 




19 What is the cost (in dollars) of 
start-up materials? 




20 What is the cost (in dollars) of 
start-up equipment? 




21 What is the cost (in dollars) of 

start-up (initial) training? 




22 What are other start-up (initial) 
implementation costs (in dollars)? 




23 Add values in rows 19 through 22. 
These are the total start-up 
implementation costs. 




24 What are the estimated annual 
costs of the intervention after 
start-up? 
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STEP 5: Summarize 


Convene the entire team and discuss the feasibility of selecting and implementing each 
intervention in your educational setting. What are the advantages and disadvantages of each 
intervention? What have you learned about the targeted interventions, based on the use of 
this tool? The response can include summarizing the evidence base and the degree to which 
the interventions align with the specific context of your state, district, or school. 


Notes 
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Section 5: Additional Resources 


This section identifies additional materials and background information to build knowledge, 
understanding, and capacity to use evidence for improvement. The resources referenced in 
this section were selected using a rubric (included at the end of this section) to ensure the 
quality, relevance, and usefulness of the materials. The types of resources include: 

• Samples of publicly available tools to support evidence-based improvement; 

• Examples of directories that list evidence-based interventions; and 

• Reports, research and case studies, and other publications. 

Each resource includes a brief description and details for accessing it. Nearly all the 
resources are publicly available and represent an initial set of materials to get you started. 
Over time, additional resources may be added, especially those that address the other steps 
in the continuous improvement cycle. Resources are organized around the following topics: 

• Every Student Succeeds Act (ESSA) guidance; 

• Evidence-based improvement (General information); 

• Examining and selecting evidence-based interventions (Step 2: Select); and 

• Clearinghouses of evidence-based interventions. 

Additionally, we encourage accessing technical assistance resources through Regional 
Educational Laboratories, Comprehensive Centers, or the State Support Network. More 
information and contact information may be found at these links: 

• Regional Educational Laboratories (RELs). The REL program, sponsored by the 
Institute of Education Sciences (IES) at the U.S. Department of Education, serves the 
education needs of designated regions, using applied research, development, 
dissemination, and training and technical assistance, to bring the latest and best 
research and proven practices into school improvement efforts. 
(www.ies.ed.gov/ncee/edLabs/reqions/ ) 

• Comprehensive Centers (CCs). The CCs include 15 regional centers that provide 
services primarily to state education agencies (SEAs) to enable them to assist school 
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districts and schools, especially low-performing schools. The CC network also 
includes seven content centers focused on specific areas of expertise such as 
turnaround, accountability, and early learning. 
(www2.ed.qov/about/contacts/qen/othersites/compcenters.html) 

• State Support Network (SSN). The SSN is a technical assistance initiative of the U.S. 
Department of Education, Office of State Support, designed to support state and 
district school improvement efforts, ( www.air.org/proiect/state-support-network-ssn) 

❖ TOPIC: ESSA Guidance and Information 

The Every Student Succeeds Act (ESSA), Reauthorization of the Elementary 
and Secondary Education Act (ESEA) of 1965 

U.S. Department of Education, 2015 

This is the legislation passed by Congress and signed by President Obama in December 
2015 that reauthorizes ESEA. 

Available at: http://www2.ed.gov/documents/essa-act-of-1965.pdf 

Non-Regulatory Guidance: Student Supports and Academic Enrichment 
Grants 

U.S. Department of Education, October 2016 

This guidance provided by the U.S. Department of Education offers information to state 
education agencies (SEAs), local education agencies (LEAs), schools, educators, and 
partner organizations information on the provisions of Title IV, Part A of ESEA, as 
amended by ESSA and how to best use funds based on these provisions. 

Available at: http://www2.ed.qov/policv/elsec/leq/essa/essassaeqrantquid1 021 201 6.pdf 

Non-Regulatory Guidance for Title II, Part A: Building Systems of Support for 
Excellent Teaching and Leading 

U.S. Department of Education, September 27, 2016 

This guidance provided by the U.S. Department of Education offers information to SEAs, 
LEAs, schools, educators, and partner organizations about how to best use funds from 
Title II, Part A of ESEA, as amended by ESSA. 

Available at: http://www2.ed.qov/policv/elsec/leq/essa/essatitleiipartaquidance.pdf 

Non-Regulatory Guidance: Using Evidence to Strengthen Education 
Investments 

U.S. Department of Education, September 16, 2016 

This guidance provided by the U.S. Department of Education offers information to SEAs, 
LEAs, schools, educators, and partner organizations about how to select and use 
“evidence-based” interventions, as defined in Title VIII of ESEA, as amended by ESSA. 

Available at: http://www2.ed.qov/policv/elsec/leq/essa/quidanceuseseinvestment.pdf 
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Supporting School Reform by Leveraging Federal Funds in a Schoolwide 
Program: Non-Regulatory Guidance 

U.S. Department of Education, September 2016 

This guidance provided by the U.S. Department of Education explains how operating a 
schoolwide program under Title I, Part A of ESEA, as amended by ESSA can be 
beneficial to LEAs and schools. 

Available at: https://www2.ed.aov/policv/elsec/lea/essa/essaswpguidance919201 6.pdf 

Better Evidence, Better Choices, Better Schools 

Steve Fleischman, Caitlin Scott, and Scott Sargrad 
American Progress, August 2016 

This report clarifies the definition of “evidence-based” that ESSA uses, distinguishing it 
from the “scientifically based research” provisions of No Child Left Behind (NCLB) and 
providing a framework for how SEAs can maximize collaborative efforts to implement 
evidence-based school improvement practices. 

Available at: 

https://www.americanprogress.org/issues/education/report/2016/08/31/143223/better-evidence- 

better-choices-better-schools 

Resources Page, Evidence in Education Lab 

Results for America 

These resources outline the potential and promise of ESSA's evidence provisions to help 
state and local leaders improve K-1 2 education. 

Available at: http://results4america.org/ed-lab-resources/ 

❖ TOPIC: Evidence-based Improvement 

Continuous Improvement In Education 

Sandra Park, Stephanie Hironaka, Penny Carver, and Lee Nordstrum 
Carnegie Foundation for the Advancement of Teaching, May 2013 

This white paper provides a preliminary view into how three educational organizations 
have undertaken continuous improvement. The paper describes the tools and 
methodology used by these organizations in their efforts. 

Available at: http://archive.carneaiefoundation.org/pdfs/elibrarv/carneaie- 
foundation continuous-improvement 2013.05.pdf 

Developing a Coherent Research Agenda Workshop 

Julie R. Kochanek, Natalie Lacireno-Paquet, and Rebecca Carey 
Regional Educational Laboratory Northeast Et Islands, July 2014 

The workshop serves as a resource for teams or groups seeking to establish a research 
focus in specific education topics. The materials include an agenda, participant 
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workbook, facilitator’s guide, and slide deck and are complemented by a 10-minute 
multimedia presentation — Developing a Research Agenda: Experiences of REL 
Northeast & Islands. 

Available at: www.relnei.org/publications/workshop-materials-for-settina-a-coherent- 
research-aaenda.html 

Education Logic Model Application (ELM) 

REL Pacific 

The ELM is a downloadable application that guides the user to create a logic model 
through a series of questions and entry of program details. 

Available at: http://relpadfic.mcrel.org/resources/elm-app 

Learning to Improve: How America’s Schools Can Get Better at Getting 
Better 

Anthony S. Bryk, Louis M. Gomez, Alicia Grunow, and Paul G. LeMahieu 
Harvard Education Press, 2015 

This book, organized around six principles, shows how a process of disciplined inquiry 
coupled with the use of networks can successfully scale up promising interventions. It 
emphasizes how “networked communities” can bring together researchers and 
practitioners to accelerate learning in key areas of education. 

Available at: www.carneaiefoundation.orq/resources/publications/learning-to-improve 

Logic Models for Program Design, Implementation, and Evaluation: 
Workshop Toolkit 

Karen Shakman and Sheila At. Rodriguez 

Regional Educational Laboratory Northeast Et Islands, May 2015 

This toolkit is designed to help practitioners learn the purpose of logic models, the 
different elements of a logic model, and the appropriate steps for developing and using a 
logic model for program evaluation. 

Available at: http://www.relnei.orq/publications/proqram-policv-evaluation-toolkit.html 

Toolkit for a Workshop on Building a Culture of Data Use 

Nancy Gerzon and Sarah Guckenburg 

Regional Educational Laboratory Northeast Et Islands, April 2015 

This field-tested workshop toolkit guides facilitators through a set of structured activities 
to develop an understanding of how to foster a culture of data use in districts and 
schools. Supporting materials — a facilitator guide and agenda, a slide deck, and 
participant handouts — provide workshop facilitators with all the materials needed to lead 
this process in their own setting. 

Available at: http://www.relnei.orq/publications/culture-data-use-toolkit.html 
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Understanding Evidence 

Centers for Disease Control and Prevention 

This is a website designed to support evidence-based decision-making. The site offers 
training to learn more about the different types of evidence, provides resources to help 
gather evidence, and presents a Continuum to discover the evidence behind an existing 
program, practice, or policy. 

Available at: http://vetoviolence.cdc.gov/apps/evidence/ 

❖ TOPIC: Examining and Selecting Evidence-Based Interventions 

What does it mean when a study finds no effects? 

Neil Seftor 

U.S. Department of Education, Institute of Education Sciences, National Center for 
Education Evaluation and Regional Assistance, October 2016 

This short brief for education decisionmakers discusses three main factors that may 
contribute to a finding of no effects: failure of theory, failure of implementation, and failure 
of research design. It provides readers with questions to ask themselves to better 
understand 'no effects' findings, and describes other contextual factors to consider when 
deciding what to do next. 

Available at: http://ies.ed.qov/pubsearch/pubsinfo.asp?pubid=REL2017265 

Discussion Tree Template 

REL Northeast Et Islands 

The Discussion Tree Template is designed to help education practitioners and 
policymakers think about the implications of research findings in their own contexts. The 
tool can be customized for the appropriate state, district, or school context. 

Available at: http://www.relnei.ora/tools-resources/discussion-tree-tool-template.html 

Evidence-Based Interventions: A Guide for States 

Livia Lam, Charmaine Mercer, Anne Podolsky, and Linda Darling-Hammond 
Learning Policy Institute, 2016 

This brief presents the research base and related conditions under which four commonly 
used interventions have been found to be effective when well-implemented. Specifically, 
these four areas are: high-quality professional development, class-size reduction, 
community schools and wraparound services, and high school redesign. 

Available at: https://learninqpolicvinstitute.org/product/evidence-based-interventions 

Evidence-Based Policymaking: A Guide for Effective Government 

The Pew Charitable Trusts and MacArthur Foundation, November 2014 

The report presents a framework for governments to build and support a system of 
evidence-based policymaking. Based on an extensive review of research and in-depth 
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interviews with government officials, practitioners, and academic experts, the framework 
identifies steps that both the executive and legislative branches can take to drive the 
development, funding, implementation, and monitoring of policies and programs. 

Available at: http://www.pewtrusts.org/en/research-and- 

analvsis/reports/2014/1 1/evidence-based-policvmakinq-a-quide-for-effective-aovernment 

School Leadership Interventions Under the Every Student Succeeds Act 

Rebecca Herman, Susan M. Gates, Emilio R. Chavez-Herrerias, and Mark Harris 
Rand Corporation, April 2016 

This report describes the levels of evidence under ESSA, and offers a synthesis of the 
research base related to those levels. The information is intended to guide policymakers 
at all levels of the education system (federal, state, and district) to use research-based 
interventions; help identify improvement activities; and support implementation of chosen 
interventions. 

Available at: www.wallacefoundation.org/knowledqe-center/Paqes/School-Leadership- 
Interventions-Every-Student-Succeeds-Act-Volume-I.aspx 

An SEA Guide for Identifying Evidence-Based Interventions for School 
Improvement 

Laurie Lee, John Hughes, Kevin Smith, and Barbara Foorman 

Florida Center for Reading Research, Florida State University, November 2016 

This guide helps SEAs to review evidence that supports the interventions that they will 
require or recommend in their state ESSA plan and funding applications. The guide also 
assists with determining levels or strength of evidence and planning for providing 
resources to LEAs. 

Available at: www.avpf.org/wp-content/uploads/2016/10/Studv-Guide-John-and-Laurie- 
Florida.pdf 

A Survey Tool for Measuring Evidence-Based Decision-Making Capacity in 
Public Health Agencies 

Julie A. Jacobs et al. 

BMC Health Services Research, 2012 

This tool provides a research-based methodology for assessing the current evidence- 
based decision-making capacity of the public health workforce. Public health agencies 
serve a wide range of populations with varying levels of resources. This survey tool 
allows an individual agency to collect data that reflects its unique workforce. This tool 
could inform a similar approach in education. 

Available at: http://bmchealthservres.biomedcentral.eom/articles/1 0.1 1 86/1 472-6963-1 2- 
57 
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Understanding Evidence: A Guide for Educators 

Gregory Chojnacki, Alexandra Resch, Alma Vigil, Ignacio Martinez, and Steve Bates 
Mathematica, Center for Improving Research Evidence, November 9, 2016 

This guide, focused on educational technologies, highlights four key types of evidence 
that educators are likely to encounter. It describes how to review claims about 
effectiveness and includes information about types of evidence ordered from weakest to 
strongest. The guide includes descriptions accompanied by examples of information 
sources containing that type of evidence. 

Available at: https://www.mathematica-mpr.com/our-publications-and- 
findinqs/publications/understandinq-types-of-evidence-a-quide-for-educators 

❖ TOPIC: Clearinghouses of Evidence-Based Interventions 

What Works Clearinghouse 

The What Works Clearinghouse (WWC) is an initiative of the Institute of Education 
Sciences, the independent, nonpartisan statistics, research, and evaluation arm of the 
U.S. Department of Education. The WWC provides educators, policymakers, 
researchers, and the public with a free, centralized source of scientific evidence on what 
works, in education, to improve student outcomes. Its goal is to provide educators and 
decisionmakers with the evidence that they need in order to make evidence-based 
decisions. The WWC focuses on the results from high-quality research to answer the 
question “What works in education?” 

Specific WWC website sections or tools that may be of particular interest to 
decisionmakers include: 

• Find What Works, a comprehensive source for information about what the WWC’s 
systematic reviews of the research say about education programs, products, 
practices, and policies, with special tools that allow users to compare interventions. 
Also, a new tool allows users to seek out information on whether research on an 
intervention has been conducted with students similar to theirs. 

• Intervention Reports, which summarize existing research on a specific program, 
product, policy, or practice, and Intervention Snapshots, which present an 
intervention in an easy-to-access format. 

• Practice Guides with recommended practices, based on an expert panel’s synthesis 
of reviews. 

• Reviews of Individual Studies, with a search tool that allows users to find individual 
studies that have been reviewed by the WWC. Search filters allow users to screen by 
topic area, study design, and WWC study rating, to create more precise evidence 
searches. 

Available at: http://www.whatworks.ed.qov 
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Best Evidence Encyclopedia (BEE) 

The Best Evidence Encyclopedia is a free website created by the Johns Hopkins 
University School of Education’s Center for Data-Driven Reform in Education (CDDRE). It 
provides summaries of scientific reviews of education interventions as well as links to the 
full text of each review. 

Available at: http://www.bestevidence.org/index.cfm 

Blueprints for Healthy Youth Development 

Blueprints for Healthy Youth Development is a registry of evidence-based youth 
development programs designed to promote the health and well-being of children and 
teens. Programs in the registry are family-, school-, and/or community-based. 

Available at: http://www.blueprintsproqrams.com 

Campbell Collaboration 

This website provides access to reviews and research syntheses to support evidence- 
based decision and policymaking. 

Available at: www.campbellcollaboration.org/ 

Education Resources Information Center (ERIC) 

ERIC is a free, online library of education research, sponsored by the Institute of 
Education Sciences (IES) of the U.S. Department of Education. It includes abstracts of 
research studies and some full-text documents. 

Available at: https://eric.ed.gov/ 

Results First Clearinghouse Database 

This database, sponsored by the Pew Charitable Trusts, contains information from eight 
national clearinghouses that conduct systematic research reviews to identify what works 
in several areas of social programs and education. 

Available at: http://www.pewtrusts.org/en/research-and-analysis/issue- 
briefs/2014/09/results-first-clearinqhouse-database 
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Evidence-Based Improvement Resources Vetting Rubric 

Each numbered element (1 , 2, 3) below the Quality, Relevance and Usefulness headings is worth up to 
three points (0=not at all, 1=slightly, 2=moderately, 3=to a great extent). To be considered for inclusion in 
the resources section of this guide, a resource must have a score of at least 15, with at least five points, 
in each of Quality, Relevance, and Usefulness, and minimally a rating of 1 for each numbered element. 
Not all sub-bullets (a, b, c) apply to all resources. These are marked as “NA”. 


Quality Relevance Usefulness 

(well-designed/developed) (is realistic and contextual) (supports implementation/use) 


1) Aligned with legal or regulatory 1) 

guidance 

a) Utilizes or references ESSA 
levels of evidence 

b) References or is consistent 
with ESSA program 
guidance 

c) References evidence-based 
improvement cycle 

2) Grounded in current research 

and practice 

a) Clearly applies and aligns 
with ESSA evidence 
standards 

b) Describes evidence of 
effectiveness with 
appropriate attention to 
research 

c) Describes emerging and 
promising practices with 
appropriate reservations 

3) Complete and purposeful 

a) Clearly and thoroughly 
communicates purpose of 
resource 

b) Structure and content are 
consistent with the purpose 

c) Specifies necessary 
expertise, experience and 
resources required 


Addresses needs 

a) Addresses common needs 
or frequently asked 
questions 

b) Aligned with SEA or LEA 
required tasks under ESSA 

c) Appropriate for ESSA 
programs or processes 

2) Potential for application 

a) Content aligned with target 
audience knowledge and 
skills 

b) Applies to or is appropriate 
for a fairly broad audience 

c) Feasible for the intended 
audience 

3) Addresses contextual factors 

a) Applicable to a variety of 
contexts (e.g., rural, urban, 
grade spans, English 
learner students) 

b) Adequately, describes 
potential barriers 

c) Includes sufficient options 
for variations 


1) Knowledge transfer 

a) Provides clear and 
comprehensive information 
to inform decisions or 
practice 

b) Uses comprehensible 
language appropriate to 
audience 

2) Usability 

a) Clearly describes 
necessary steps, conditions 
and resources for 
implementation 

b) Comprehensive coverage 
of topic or refers to other 
resources 

3) Applicability 

a) Includes sufficient case 
studies, examples, or 
possible variations 

b) Includes specific 
information about or 
considerations for how to 
plan, implement and 
monitor activity 
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